Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker agents are not launched on ci.jenkins.io #4359

Closed
MarkEWaite opened this issue Oct 16, 2024 · 5 comments
Closed

Docker agents are not launched on ci.jenkins.io #4359

MarkEWaite opened this issue Oct 16, 2024 · 5 comments

Comments

@MarkEWaite
Copy link

Service(s)

ci.jenkins.io

Summary

Jobs that use an agent with the label docker have not launched on ci.jenkins.io for several hours. https://ci.jenkins.io/job/Core/job/acceptance-test-harness/job/PR-1788/1/ is an example of a job that is blocked waiting for the docker-highmem label

Reproduction steps

  1. Check jobs in the ci.jenkins.io build queue, note that many of those jobs are waiting for an agent with the docker label or the docker-highmem label
@MarkEWaite MarkEWaite added the triage Incoming issues that need review label Oct 16, 2024
@dduportal dduportal added this to the infra-team-sync-2024-10-22 milestone Oct 16, 2024
@dduportal dduportal self-assigned this Oct 16, 2024
@dduportal dduportal removed the triage Incoming issues that need review label Oct 16, 2024
@dduportal
Copy link
Contributor

Looks like an unintended consequence of #4358 (comment).

We are again caught in the Azure VM plugin not refreshing credential :'(

Just restarted ci.jenkins.io and agents are spawning again. Let's amend the procedure used for #4358

@dduportal
Copy link
Contributor

Temporarily increased the maximum size of cloud agents: 100 each (instead of 50) to absorb the build wave

@dduportal
Copy link
Contributor

Build queue is emptied: builds are being performed.

@dduportal
Copy link
Contributor

Note: the PR jenkins-infra/azure#854 DID mention a restart at the end:

then restart the controller to ensure that the old credential is not kept in cache.

=> I'm the culprit here. Sorry for the blocker :(

@krisstern
Copy link
Member

Thanks @dduportal for the prompt response!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants