k/record_batcher: Move to kafka/data #23832
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Useful in audit_log_manager as well as transform logging Signed-off-by: Oren Leiman <[email protected]>
And adds make_batch_of_one Signed-off-by: Oren Leiman <[email protected]>
Signed-off-by: Oren Leiman <[email protected]>
We need these for sorting out which partitions are locally led Signed-off-by: Oren Leiman <[email protected]>
Previous implementation used a very high value for retries on the internal kafka client, which prevents the client from recovering certain types of errors. Instead, we batch up drained records on the manager side, allowing us to hold a copy of each batch in memory and retry failed produce calls from "scratch". This also allows us to be _much_ more aggressive about batching. The internal kafka client will calculate a destination partition for each record, round robin style over the number of partitions. In the new scheme, we shoot for a maximally sized batch first, then select a destination, still round-robin style, but biasing heavily toward locally led partitions. In this way, given the default audit per-shard queue limit and default max batch size (both 1MiB), the most common drain operation should result in exactly one produce request. Signed-off-by: Oren Leiman <[email protected]>
github-actions
bot
added
area/build
area/redpanda
area/wasm
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Followup to #23775
Backports Required
Release Notes