-
Notifications
You must be signed in to change notification settings - Fork 4.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Scan all status is hung after migration from 1.8 to 1.10 #9963
Comments
en? This is very weird. Total is 0 but ongoing is true. @AllForNothing which property you're using to disable the button? |
on
ongoing |
Let's be clear that this issue is because the Harbor is shutdown while the job is running and the interrupted job is blocked after Harbor is started. So it does not only impact upgrading. |
@reasonerjt Yes, it's upgrade regardless. |
I think the issue can be generalized to the case that, if there is a Redis restart when lots of jobs are in progress, there might be cause job loss issues. Once the jobs could not be recovered, the status data in the database will be out of sync and look like |
Could we add release notes for such known issues?
|
@steven-zou yes of course. Added the |
@steven-zou we also need to mention the workaround when this happens. |
what's the workaround and is it something that happens automatically or someone has to manually execute it? |
does this "hung" issue happen 100% on each upgrade? |
No. It will happen only and only if there are jobs are running and then Redis restart occurs for any reason. Most of the time, when users doing their upgrading, there will not be jobs running. I'll continue to do investigate this issue. The possible root cause is data loss happens after Redis restarts. Current plan is we can deliver a fix in the patch release 1.10.X. |
The workaround approaches will be noted in the FAQ list. By manually triggering API calls. |
comments after discussion: i think we are fine to release with this ticket and fix it in the next patch release. this is not data loss, because Harbor is eventually consistent based on user intent. A redis cache restart (for whatever reasons) will chase the same effects and Harbor will be eventually consistent. |
Added to the RNs, so removing
|
It may be related to this issue in the upstream project : gocraft/work#146 |
- improve the status hook sending/resending approach - improve the status compare and set approach - simplify the relevant flow - add reaper to fix the out of sync jobs - fix goharbor#10244 , fix goharbor#9963 Signed-off-by: Steven Zou <[email protected]>
- improve the status hook sending/resending approach - improve the status compare and set approach - simplify the relevant flow - add reaper to fix the out of sync jobs - fix goharbor#10244 , fix goharbor#9963 Signed-off-by: Steven Zou <[email protected]>
- improve the status hook sending/resending approach - improve the status compare and set approach - simplify the relevant flow - add reaper to fix the out of sync jobs - fix goharbor#10244 , fix goharbor#9963 Signed-off-by: Steven Zou <[email protected]>
- improve the status hook sending/resending approach - improve the status compare and set approach - simplify the relevant flow - add reaper to fix the out of sync jobs - fix goharbor#10244 , fix goharbor#9963 Signed-off-by: Steven Zou <[email protected]>
- improve the status hook sending/resending approach - improve the status compare and set approach - simplify the relevant flow - add reaper to fix the out of sync jobs - fix goharbor#10244 , fix goharbor#9963 Signed-off-by: Steven Zou <[email protected]>
- improve the status hook sending/resending approach - improve the status compare and set approach - simplify the relevant flow - add reaper to fix the out of sync jobs - fix goharbor#10244 , fix goharbor#9963 Signed-off-by: Steven Zou <[email protected]>
- improve the status hook sending/resending approach - improve the status compare and set approach - simplify the relevant flow - add reaper to fix the out of sync jobs - fix goharbor#10244 , fix goharbor#9963 Signed-off-by: Steven Zou <[email protected]>
The code work is done, pending for PR review. Move to next SP 78 |
- improve the status hook sending/resending approach - improve the status compare and set approach - simplify the relevant flow - add reaper to fix the out of sync jobs - fix goharbor#10244 , fix goharbor#9963 Signed-off-by: Steven Zou <[email protected]>
* fix[jobservice]:job status is hung after restart - improve the status hook sending/resending approach - improve the status compare and set approach - simplify the relevant flow - add reaper to fix the out of sync jobs - fix goharbor#10244 , fix goharbor#9963 Signed-off-by: Steven Zou <[email protected]> * add content trust middleware in new v2 handler * fix * add policy check middleware in v2 handler * merge with latest * fix Co-authored-by: Steven Zou <[email protected]>
- improve the status hook sending/resending approach - improve the status compare and set approach - simplify the relevant flow - add reaper to fix the out of sync jobs - fix goharbor#10244 , fix goharbor#9963 Signed-off-by: Steven Zou <[email protected]>
- improve the status hook sending/resending approach - improve the status compare and set approach - simplify the relevant flow - add reaper to fix the out of sync jobs - fix goharbor#10244 , fix goharbor#9963 Signed-off-by: Steven Zou <[email protected]>
- improve the status hook sending/resending approach - improve the status compare and set approach - simplify the relevant flow - add reaper to fix the out of sync jobs - fix goharbor#10244 , fix goharbor#9963 Signed-off-by: Steven Zou <[email protected]>
Test Steps:
1. Deploy harbor 1.8.5;
2. Set Scan all shedule to 12 12 12 * * *;
3. Wait more than 12 minutes;
4. Update harbor to 1.10;
5. View page of "Interrogation Services";
Result:
Page is in scan waiting status for long time.
The text was updated successfully, but these errors were encountered: