fix(appset): Retry on conflict when updating status #19663
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# Context: When updating the status of the applicationset object it can happen that it fails due to a conflict since the resourceVersion has changed due to a different update. This makes the reconcile fails and we need to wait until the following reconcile loop until it updates the relevant status fields and hope that the update calls don't fail again due a conflict. It can even happen that it gets stuck constantly due to this erriors. A better approach I would say is retrying when there is a conflict error with the newest version of the object, so we make sure we update the object with the latest version always. This has been raised in issue argoproj#19535 that failing due to conflicts can make the reconcile not able to proceed. # What does this PR? - Wraps all the `Update().Status` calls inside a retry function that will retry when the update fails due a conflict. - Adds appset to fake client subresources, if not the client can not correctly determine the status subresource. Refer to: kubernetes-sigs/controller-runtime#2386, and kubernetes-sigs/controller-runtime#2362. Signed-off-by: Carlos Rejano <[email protected]>
❌ Preview Environment deleted from BunnyshellAvailable commands (reply to this comment):
|
❌ Preview Environment deleted from BunnyshellAvailable commands (reply to this comment):
|
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #19663 +/- ##
=========================================
Coverage ? 55.83%
=========================================
Files ? 320
Lines ? 44251
Branches ? 0
=========================================
Hits ? 24709
Misses ? 16980
Partials ? 2562 ☔ View full report in Codecov by Sentry. |
gdsoumya
requested changes
Aug 28, 2024
gdsoumya
reviewed
Aug 29, 2024
| if err := r.Get(ctx, namespacedName, applicationSet); err != nil { | ||
| if client.IgnoreNotFound(err) != nil { | ||
| return nil | ||
| err := retry.RetryOnConflict(retry.DefaultRetry, func() error { |
There was a problem hiding this comment.
can you note down how many times it's going to retry and for how long?
There was a problem hiding this comment.
Sure, in this case it's using this backOff defined:
var DefaultRetry = wait.Backoff{
Steps: 5,
Duration: 10 * time.Millisecond,
Factor: 1.0,
Jitter: 0.1,
}
We would be doing around 5 calls every 10 ms.
Should I add it as a comment in the code? Or?
gdsoumya
approved these changes
Sep 7, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Context:
When updating the status of the applicationset object it can happen
that it fails due to a conflict since the resourceVersion has changed
because of a previous update. This makes the reconcile fails and
we need to wait until the following reconcile loop updates the
relevant status fields and hope that the update calls don't fail again
due to a conflict. It can even happen that it gets stuck constantly due
to these erriors.
A better approach I would say is retrying when there is a conflict
error with the newest version of the object, so we make sure we update
the object with the latest version always.
This has been raised in issue #19535 that failing due to conflicts can
make the reconciliation not able to proceed.
What does this PR?
Update().Statuscalls inside a retry function that will retry when the update fails due to a conflict.