Migrate to the official Zeebe client and standalone Operate client

[chillleader]

What should we do?

Operate client is being moved to a separate repository, and the community-supported Spring Zeebe is being sunset.
We need to switch to the official Zeebe client and standalone Operate client.

Operate client: https://github.com/camunda-community-hub/camunda-operate-client-java

Specifically, we need to check that all configurations work as before:

• Properties files in the connectors-bundle
• Docker-compose files form camunda-platform
• Helm charts
• SaaS config
• Console Secret API client (authentication depends on the classes from Spring Zeebe)

Why should we do it?

To use supported and up-to-date versions of dependencies.

[chillleader]

Operate client was released: https://github.com/camunda-community-hub/camunda-operate-client-java/releases/tag/8.5.8-rc1

[chillleader]

We also need the new testing library to replace spring-boot-starter-camunda-test from the community project.

It has been merged already but not yet released: camunda/camunda#18999

[akeller]

How confident are you that this will be delivered with 8.6 or an upcoming alpha? It's currently blocking me here - camunda/camunda-docs#4065 (comment)

[chillleader]

I prepared the migration in this PR: #3193
It's mostly ready and tested against the local environment, however, some of the e2e tests still need fixes which couldn't be implemented quickly.

Given that code freeze is this week, unfortunately I don't think it's feasible to include it in the upcoming alpha, the best timing from my point of view would be 8.7-alpha1.

Here are some outcomes:

• The property structure will change
• For SaaS, we need to keep the backward compatibility with previous versions that run on community code and use a different property structure
• We will need to modify Helm charts to support the new property structure there
• I re-implemented the Console secrets client to use the authentication code from the new Operate client which is very similar to the old approach in Spring Zeebe
• The new Spring-based testing library from the monorepo is still under construction, there are a few things that make using it right now uncomfortable for us:
  • Limited number of assertions supported compared to the community version, the process variable assertions are very important for us but they will only be added in the upcoming alpha and for now we would have to use the snaphot/rc versions in our release
  • All tests are now very long-running as there is no embedded Zeebe engine like before; all tests now spin up the whole Camunda cluster even though we only need Zeebe.

[aleksander-dytko]

@chillleader thanks for the update!
Do I understand correctly, that we shouldn't archive the community Zeebe Client yet with 8.6?

[chillleader]

Yes, I think we should wait until at least 8.7-alpha1 to archive it.

I think prior to archiving it, we'll also need to agree how to deal with the older Connectors versions that will continue using the community client after it's archived, i.e. what should we do if we discover a bug in Spring Zeebe 8.4 and need to provide a fix for Connectors. I'll discuss this with the team

cc @sbuettner