Support smooth transition to another topic/records-storage (virtual topics?)

[lanwen]

Basically, we need a quick way to define a new destination only on liiklus side and get it working without changes on the service side.

Usecase:

• 20 microservices pointing to one topic
• GDPR cleanup of that topic with name change and offsets change on the records storage side
• Or fail of the storage site with recreation from the ground with the different offsets

Need a way to actualize the consumer's targets.

How we deal with it right now:

• Setup one more liiklus with another db
• Point dumb consumer to that liiklus and read the new topic
• Stop reads and replicate in the new db all consumer groups with numbers from the dumb consumer
• Restart all liiklus instances with the new db
• Start writes

What idea do I have:

• Management api with authenticated rest call to fetch the latest offset from the records storage and write to a specific consumer group
• Somehow map on the records storage side where to get real destination and what are real offsets there (see prev step)

Basically, if someone comes with farmers-v1 group for the events topic, actually treat that as persistent://europe/apples/markets-q1 topic of pulsar. And after the call to that api - next message will come from the persistent://america/fruits/veggies topic with other offsets.

Currently, can see only one option to implement that - combine dynamo and metrics plugins and extend db structure to keep that rewrite info.

Maybe keep not only topic rewrite but the destination of the records as well, in case we want to migrate from kafka to pulsar or vice-versa.