Add ConcurrentBrokerRestartCountPerRack to RollingUpgradeConfig

[ctrlaltluc]

Description

Add ConcurrentBrokerRestartCountPerRack property to RollingUpgradeConfig, in order to allow multiple brokers to be restarted in parallel during a rolling upgrade.

This PR is a prerequisite for #1001.

Type of Change

• New Feature

Checklist

• I have read the contributing guidelines
• Existing issues have been referenced (where applicable)
• I have verified this change is not present in other open pull requests
• Functionality is documented
• All code style checks pass
• New code contribution is covered by automated tests
• All new and existing tests pass

[bartam1]

Just few comments. Otherwise LGTM.
Thank you for the contribution!

[ctrlaltluc]

@bartam1 Hi Marton, thanks a lot for the thorough reviews! Was on holiday until now. All your comments make sense, will have a look on your suggestions on the other PR too. Addressed all comments on this one.

[ctrlaltluc]

Rebased again with master, ready for final review.

[bartam1]

LGTM!

[ctrlaltluc]

@panyuenlau this PR requires one more review to be merged. Can you help please? Asking you as you already had a look on #1001 and this one is a prerequisite for that. Thanks! :)

[panyuenlau]

reviewed the main implementation - looks great

[ctrlaltluc]

Rebased again with master (no other changes) in hopes to have it merged and there is an integration test failing. First time I see it failing here.
Probably a flaky one? Is there any way to re-run it?

[FAILED] [120.019 seconds]
KafkaClusterNodeportExternalAccess [JustBeforeEach] when NodePortExternalIP is configured reconciles the service successfully
  [JustBeforeEach] /home/runner/work/koperator/koperator/controllers/tests/kafkacluster_controller_externalnodeport_test.go:78
  [It] /home/runner/work/koperator/koperator/controllers/tests/kafkacluster_controller_externalnodeport_test.go:147

[panyuenlau]

Rebased again with master (no other changes) in hopes to have it merged and there is an integration test failing. First time I see it failing here. Probably a flaky one? Is there any way to re-run it?

[FAILED] [120.019 seconds]
KafkaClusterNodeportExternalAccess [JustBeforeEach] when NodePortExternalIP is configured reconciles the service successfully
  [JustBeforeEach] /home/runner/work/koperator/koperator/controllers/tests/kafkacluster_controller_externalnodeport_test.go:78
  [It] /home/runner/work/koperator/koperator/controllers/tests/kafkacluster_controller_externalnodeport_test.go:147

Context: The controller test failure "spec.ports[0].nodePort: Invalid value: 31125: provided port is already allocated" is seen from time to time in GHA

My guess: this is probably because the underlying node that GHA uses to run the tests happens to be the same one that was used in previous run, and GHA's infrastructure didn't get to clean up the allocated port in time

[ctrlaltluc]

Rebased again with master (no other changes) in hopes to have it merged and there is an integration test failing. First time I see it failing here. Probably a flaky one? Is there any way to re-run it?

[FAILED] [120.019 seconds]
KafkaClusterNodeportExternalAccess [JustBeforeEach] when NodePortExternalIP is configured reconciles the service successfully
  [JustBeforeEach] /home/runner/work/koperator/koperator/controllers/tests/kafkacluster_controller_externalnodeport_test.go:78
  [It] /home/runner/work/koperator/koperator/controllers/tests/kafkacluster_controller_externalnodeport_test.go:147

Context: The controller test failure "spec.ports[0].nodePort: Invalid value: 31125: provided port is already allocated" is seen from time to time in GHA

My guess: this is probably because the underlying node that GHA uses to run the tests happens to be the same one that was used in previous run, and GHA's infrastructure didn't get to clean up the allocated port in time

@panyuenlau understood.

We thought it only happens in the Adobe fork.
We attempted to fix this in adobe#67, by explicitly deleting the nodeports. I thought it was related to envtest not doing garbage collection (when KafkaCluster is deleted, even if it is a parent of those nodeports, the nodeports are not deleted):

One common example is garbage collection; because there are no controllers monitoring built-in resources, objects do not get deleted, even if an OwnerReference is set up; as a consequence, usually test implements code for cleaning up created objects.

But it seems explicitly deleting the nodeports does not fix it either. We still saw it afterwards, on occasion.