Rewrite Java API Table.readJSON to return the output from libcudf read_json directly
#17180
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ttnghia
added
feature request
revans2
approved these changes
Oct 25, 2024
There was a problem hiding this comment.
I love this change. My biggest problem is that this is a breaking change.
I put in a number of changes into the Spark Plugin to provide the emptyRowCount and fully removing those changes is not simple. So either please make sure that this provides some backwards compatibility so we can make the change in a few steps or can we please have the other PR ready to go so that there is very little down time between the two.
This reverts commit a82fdb699a13008b878deaab18ae85a440cf05af.
Signed-off-by: Nghia Truong <[email protected]>
revans2
approved these changes
Oct 25, 2024
ttnghia
added
non-breaking
|
Changed to non-breaking as the old Java methods are not removed in this PR. We can remove them later on when all the plugin code complete their adaptation. |
3 tasks
|
After this (with #17029), the overhead of reordering columns is significantly reduced (above is before this, and below is with this):
|
revans2
approved these changes
Oct 25, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
With this PR,
Table.readJSONwill return the output from libcudfread_jsondirectly without the need of reordering the columns to match with the input schema, as well as generating all-nulls columns for the ones in the input schema that do not exist in the JSON data. This is because libcudfread_jsonalready does these thus we no longer have to do it.Depends on:
Partially contributes to NVIDIA/spark-rapids#11560.
Note that since the Java API
Table.readJSONno longer needs to generate all-nulls columns, the function signature is changed and will cause spark-rapids build to break.