Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add new nvtext minhash_permuted API #16756

Draft
wants to merge 72 commits into
base: branch-24.12
Choose a base branch
from

Conversation

davidwendt
Copy link
Contributor

@davidwendt davidwendt commented Sep 5, 2024

Description

Introduce new nvtext minhash API that takes a single seed for hashing and 2 parameter vectors to calculate the minhash results from the seed hash:

std::unique_ptr<cudf::column> minhash_permuted(
  cudf::strings_column_view const& input,
  uint32_t seed,
  cudf::device_span<uint32_t const> parameter_a,
  cudf::device_span<uint32_t const> parameter_b,
  cudf::size_type width,
  rmm::cuda_stream_view stream ,
  rmm::device_async_resource_ref mr);

The seed is used to hash the input using rolling set of substrings width characters wide.
The hashes are then combined with the values in parameter_a and parameter_b to calculate a set of 32-bit (or 64-bit) values for each row. Only the minimum value is returned per element of a and b when combined with all the hashes for a row. Each output row is a set of M values where M = parameter_a.size() = parameter_b.size()

This implementation is significantly faster than the current minhash which computes hashes for multiple seeds.

Included in this PR is also the minhash64_permuted() API that is identical but uses 64-bit values for the seed and the parameter values. Also included are new tests and a benchmark as well as the pylibcudf and cudf interfaces.

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@davidwendt davidwendt added 2 - In Progress Currently a work in progress libcudf Affects libcudf (C++/CUDA) code. improvement Improvement / enhancement to an existing function non-breaking Non-breaking change labels Sep 5, 2024
@davidwendt davidwendt self-assigned this Sep 5, 2024
@davidwendt davidwendt changed the base branch from branch-24.10 to branch-24.12 September 23, 2024 23:25
@davidwendt davidwendt changed the title Improve minhash performance by using more working memory Add new nvtext minhash_permuted API Oct 24, 2024
rapids-bot bot pushed a commit that referenced this pull request Oct 25, 2024
Deprecates the current nvtext minhash functions some of which will be replaced in #16756 with a different signature. The others will no longer be used and removed in future release. The existing gtests and benchmarks will be retained for rework in the future release as well.

Authors:
  - David Wendt (https://github.com/davidwendt)

Approvers:
  - Nghia Truong (https://github.com/ttnghia)
  - Bradley Dice (https://github.com/bdice)

URL: #17152
@github-actions github-actions bot added the CMake CMake build issue label Oct 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2 - In Progress Currently a work in progress CMake CMake build issue improvement Improvement / enhancement to an existing function libcudf Affects libcudf (C++/CUDA) code. non-breaking Non-breaking change pylibcudf Issues specific to the pylibcudf package Python Affects Python cuDF API.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant