Client-Server compatibility matrix

[f-ld]

For quite some time I've been working on Pulsar with the following assertion:

• clients and server need to be running the same version
• server updates has to be done of one minor version at a time (so from 2.7.x we can only go to 2.8.x and not 2.9.x or more)
• during updates, server has to be updated first and then clients (so from 2.7.x everywhere, update cluster to 2.8.x and then pulsar clients to 2.8.x).

But I've been recently reading things about unexpected compatibility issues between more a minor version difference >= 2 (example) or bigger version jumps (example) or newer client version than server (talking for example of client 2.10.x on server 2.7.x) and I'm getting lost.

Obviously, my initial assumption should be working. But am I losing time doing many small jumps while I can do fewer bigger ones?

For example, java client SDK 2.7x is causing load issues on producers doing batching under some conditions: can we skip to client 2.8.x even if servers are still on 2.7.x? And when updating servers 2.7.x, may I simply jump to 2.10.x or 2.11.x?

Note: I am talking about production grade system without possible downtime.

Is this sentence on this pulsar upgrade doc page meaning that I could do the few big jumps:

Currently, Apache Pulsar is compatible between versions.

If so, why would the same page mention clients in the upgrade sequence (and clients being last item)?

Thank you for any doc pointer because I am loosing faith in what I thought I knew.

[tisonkun]

cc @RobertIndie @BewareMyPower @momo-jun since you're working on https://pulsar.apache.org/client-feature-matrix/

[fabien-dlb]

From @david-streamlio message here in #19552, then it seems we should not have to care about geo replication between 2.7.x and 2.11.x during a rolling update.

Would it be the same inside a given cluster for everything (zookeeper, bookeeper) while performing that rolling update?

And what about clients: can clients running 2.7.x work perfectly with a cluster migrated to 2.11.x? Or would the opposite be better, have clients updated to 2.11.x while still on 2.7.x and then update the backend services to 2.11.x?

[fabien-dlb]

Some more update reading annoucement of pulsar 3.0.0:

Before Pulsar 3.0, upgrades should be performed linearly through each feature version. For example, when upgrading from 2.8 to 2.10, it is important to upgrade to 2.9 before going to 2.10.

So this confirms that we need to have brokers going through all versions and I understand that from 2.7.x we need to go through 2.8.x on broker side then client side, then to 2.9.x on broker side then client side, etc ... until 2.11.x from which version only we can update to 3.0.0 (note: still looking for confirmation that we can update from 2.x to 3.x and from what 2.x version(s) it is possible).

[tdkeeley]

I'd like to get a definitive answer to this question. The answer below suggests a client can be at a later version than the server:
#22259

I'm contemplating updating my client to 3.x, but don't have control of the server, which would remain at 2.9.4. It sounds like maybe version interoperability hasn't always been as robust as it is now, though.