Useful in audit_log_manager as well as transform logging

Signed-off-by: Oren Leiman <[email protected]>

And adds make_batch_of_one

Signed-off-by: Oren Leiman <[email protected]>

Signed-off-by: Oren Leiman <[email protected]>

We need these for sorting out which partitions are locally led

Signed-off-by: Oren Leiman <[email protected]>

Previous implementation used a very high value for retries on the
internal kafka client, which prevents the client from recovering
certain types of errors.

Instead, we batch up drained records on the manager side, allowing
us to hold a copy of each batch in memory and retry failed produce
calls from "scratch".

This also allows us to be _much_ more aggressive about batching.
The internal kafka client will calculate a destination partition
for each record, round robin style over the number of partitions.
In the new scheme, we shoot for a maximally sized batch first, then
select a destination, still round-robin style, but biasing heavily
toward locally led partitions. In this way, given the default audit
per-shard queue limit and default max batch size (both 1MiB), the
most common drain operation should result in exactly one produce
request.

Signed-off-by: Oren Leiman <[email protected]>

Followup to redpanda-data#23775