Initiate S3 Multi-part upload on receiving first event #318
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…, and closes the file on flush. Signed-off-by: Aindriu Lavelle <[email protected]>
…allowing changelog records to initiate multipart upload. Signed-off-by: Aindriu Lavelle <[email protected]>
aindriu-aiven
force-pushed
the
aindriu-aiven/initiate-multi-part-upload
branch
from
b6cebcc
to
4776d6d
Compare
aindriu-aiven
commented
Oct 25, 2024
|
|
||
| assertThat(expectedBlobs).allMatch(blobName -> testBucketAccessor.doesObjectExist(blobName)); | ||
|
|
||
| assertThat(testBucketAccessor.readLines("prefix-topic0-0-00000000000000000012", compression)) |
There was a problem hiding this comment.
As an FYI, the S3MockApi does not create the file names correctly for key, value
gharris1727
reviewed
Oct 25, 2024
There was a problem hiding this comment.
Wow, the S3OutputStream has had multipart upload for a long time: Aiven-Open/s3-connector-for-apache-kafka#73
But we were still buffering data as records, rather than offloading them early? Crazy. Thanks for the improvement.
Comment on lines
+99
to
+100
| * This determines if the file is key based, and possible to change a single file multiple times per flush or if | ||
| * it's a roll over file which at each flush is reset. |
There was a problem hiding this comment.
Can you explain more about this? What is key based grouping, and why does it mutate the file?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This update initiates the multipart upload as soon as a record begins, and closes the file on flush.
This PR does
This PR does not