Investigate how to integrate better with releases

[QuentinBisson]

Motivation

See this for context giantswarm/releases#1459 (comment)

Our current config mechanism broke releases really bad. We need to find a better way forward because there are currently no way to be sure we do not break releases when doing changes in the operators.

Let's investigate what we really miss in releases today and fix it instead of being out of process

Todo

• Investigate what configuration we actually need to be dynamic / regularly adapt (like secrets, etc)
  • create a list of configuration that our operators are actually handling
• Investigate what configuration we want to keep dynamic as a safeguard for fast reaction times on incidents
• Investigate how we technically can push configuration
  • this can be the observability bundle - but since we want to get rid of it some day, is it still the best place?
  • since the MC overrides some configuration in the WCs, this might be tricky.

Open Questions to clear with tenet

• How do we configure apps in releases? Maybe through the Observability Bundle, but that has it's own problems?
• How can we get auto patches? Since we don't want PRs every night and have a lot of blockers, just for config changes?
• There will be some dynamic config - how do we handle those changes?

Outcome

• We have a better understanding how we can better follow the given release process

[AverageMarcus]

@QuentinBisson Would you be able to outline in this issue what features y'all are missing when it comes to Releases and what your current pain points are? I think that'd really help to focus the ideas.

Some things I'd like to know more about - is the problems you're seeing more related to getting changes out to MC (where we've typically been really slow at rolling out update, which is maybe what we need to fix) or, for example, are you seeing problems getting changes out to existing workload clusters that are needed to build consistency across out platform? (These are just examples to get the thoughts flowing 😉)

[QuentinBisson]

@AverageMarcus you posted when we started refining this :D @Rotfuks added some things at the top but we are starting a discovery :)

[Rotfuks]

Thanks for looking into this issue @AverageMarcus ! We definitely have on the list to talk to the lovely people of tenet for this issue - but we identified some preparation we can first do, to do you guys justice. So we'll reach out after some initial investigation as soon as we're confident we can answer all relevant questions at this stage.