Parquet reader list microkernel #16538
Open
+584
−112
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…lable column code
…attione-nvidia/cudf into mukernels_fixedwidth_optimize
Co-authored-by: nvdbaranec <[email protected]>
Co-authored-by: nvdbaranec <[email protected]>
Co-authored-by: nvdbaranec <[email protected]>
…attione-nvidia/cudf into mukernels_fixedwidth_optimize
Co-authored-by: Vukasin Milovanovic <[email protected]>
…idia/cudf into parquet_list_kernel
|
Seems like this is also adding list support to the split page path as well. Am I reading this right? |
|
One thing I've been thinking about is maybe splitting this file into two or three pieces.
Definitely not for this PR, but something to think about down the road. I think it might help make the volume of code that has built up here more tractable. |
nvdbaranec
requested changes
Oct 17, 2024
Co-authored-by: nvdbaranec <[email protected]>
Yes. |
mythrocks
reviewed
Oct 22, 2024
vuule
reviewed
Oct 22, 2024
nvdbaranec
requested changes
Oct 25, 2024
…idia/cudf into parquet_list_kernel
vuule
approved these changes
Oct 25, 2024
nvdbaranec
approved these changes
Oct 25, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR refactors fixed-width parquet list reader decoding into its own set of micro-kernels, templatizing the existing fixed-width microkernels. When skipping rows for lists, this will skip ahead the decoding of the definition, repetition, and dictionary rle_streams as well. The list kernel uses 128 threads per block and 71 registers per thread, so I've changed the launch_bounds to enforce a minimum of 8 blocks per SM. This causes a small register spill but the benchmarks are still faster, as seen below:
DEVICE_BUFFER list benchmarks (decompress + decode, not bound by IO):
run_length 1, cardinality 0, no byte_limit: 24.7% faster
run_length 32, cardinality 1000, no byte_limit: 18.3% faster
run_length 1, cardinality 0, 500kb byte_limit: 57% faster
run_length 32, cardinality 1000, 500kb byte_limit: 53% faster
Compressed list of ints on hard drive: 5.5% faster
Sample real data on hard drive (many columns not lists): 0.5% faster
Checklist