[INFRA-3100] Migrate updates.jenkins.io to another Cloud

[jenkins-infra-bot]

Why

Read the EPIC (aws cost decrease: #2646)

What

• updates.jenkins.io serves the JSON of the update center, but it is not behind a CDN (need to be updated regularly) causing a lot of outbound data transfer (2.4 Mb for a JSON download, esitmation around 300 Gb of data sent daily, estimation around 10 Tb per month which costs some $$$.

• The VM pkg.origin.jenkins.io is managing multiple services and separting them could be a great help.

How

Different paths can be taken here: to be discussed in infra meeting + validated by the board as it's an important service.

• Migrating the service to Oracle Cloud:
  • the first 10 Tb of outbound data is free, and if the worst case, 50 Tb of outbound would cost less than 3k $ per month - https://www.oracle.com/be/cloud/networking/networking-pricing.html
  • ARM machines: cheaper and better performance for a simple webserver + SSH + data
  • Block storage: close to archives.jenkins.io
  • Risk: Oracle sponsoring program is only 1 year, and is not easily secured for paiment as other account

• Migrating to Azure
  • Easily in AKS: easier management
  • Safer paiment process
  • Risk: cost of outbound to be evaluated

---

Originally reported by dduportal, imported from: Migrate updates.ci.jenkins.io to another Cloud

• status: Open
• priority: Major
• resolution: Unresolved
• imported: 2022/01/10

[note]

• Check that Claim that updates.jenkins.io certificate is sometimes expiring #3017 is not happening once migration is finished.
• The error curl: (16) Error in the HTTP2 framing layer should not happen anymore with the latest Apache versions

[dduportal]

Requires setting up the Oracle Terraform project, like #2682

Todo list:

• Specify a new infra (VM + data storage of the same size as the actual pkg.origin.jenkins.io machine + any other Oracle infra requirements). Implies checking the pricing to get an overall idea
• Once VM is created, add it to puppet management: new node, with the role / profiles associated to "update.jenkins.io"
• Rsync the data one time
• Update the jenkins-infra/update-center2 's associated job in trusted.ci to update both "pkg.origin.jenkins.io" VM and the new one (⚠️ inline pipeline, don't search for a Jenkinsfile as code)
• Validate the new instance (including checking with Daniel, Tim and other contributors)
• Communicate about the upcoming change
• As proposed by Stephane, use a round-robin DNS record value to split traffic between old/new during some time and check for error

[dduportal]

Blocked by #2973

[smerle33]

actual machine size is :
32Gb RAM
8cpu
1,2Tb data disk (372Gb free)

about half of the power is used currently (checked with the local SAR probe)

[smerle33]

Infra to specify :

• 1 VM (similar to archive.jenkins.io)
• 1 network ("mirrors") + 1 subnet ("20210630-1531")
• 1 volume 1,2Tb
• 1 set of security groups to restrict network access
• 1 ssh key pair

[smerle33]

VM specifications :

• 4vCPU/16Gb RAM : half of actual machine
• proposal to use ARM like archive.jenkins.io --> 4ocpu (VM Type Standard A1 Flex)
• Image Ubuntu 20.04 (FULL [non minimal]) --> upgrade from ubuntu 18.04 for actual machine
  • option for 22.04 but need testing for puppet agent (+ openssl v3)

[dduportal]

Fifth brownout finished! (ref. jenkins-infra/azure-net#309)

[dduportal]

Fifth brownout finished! (ref. jenkins-infra/azure-net#309)

Results of the fifth brownout:

• ✅ The 2 last blocker issues did not come back (both HTTP/404 of different kinds)
• ✅ Still working for the main usages (JSON files, base HTML pages)
• Metrics shows the same usage on Cloudflare (reminder we did not have access to detailed logs):

Click to view Cloudflare aggregated metrics

• ⚠️ We still have a lot of HTTP/404 answers. Rate decreased a bit but it is still more than half of the requests received by Cloudflare. => not a blocker but annoying

• ✅ The successful requests on Cloudflare maps to the current Update Center (~ 3M hits for ~2.5 Tb downloaded)

• ⚠️ We still have user-facing reported HTTP/404 on the "directory listing" pages (e.g. requests to a directory without an index.html). => not a blocker but annoying

  • Apache (in the current VM pkg) has directory listing and generates the index for us.
  • While in the new system, Apache redirects these requests to the cloudflare mirrors which do not provide directory listing feature so ends in HTTP/404.
  • See comment below for details

• ⌛️ The update center2 generation jobs runs in 5 to 7 min, but is only cron-triggered every 10 min. We should work on this. => not a blocker but annoying

• ⚠️ The update center2 should copy the Apache www-redirects content as the last stage. Not important or blocker, but avoids the 3-4 min time window with HTTP/404 errors when a new update center JSON versions is deployed (usually during a new Core release). => not a blocker and low priority

[dduportal]

⚠️ We still have user-facing reported HTTP/404 on the "directory listing" pages (e.g. requests to a directory without an index.html). => not a blocker but annoying

* Apache (in the current VM `pkg`) has directory listing and generates the index for us.

* While in the new system, Apache redirects these requests to the cloudflare mirrors which do not provide directory listing feature so ends in HTTP/404.

* See comment below for details

• We were already aware of this behavior as per the past discussions
  • [INFRA-3100] Migrate updates.jenkins.io to another Cloud #2649 (comment)
  • [INFRA-3100] Migrate updates.jenkins.io to another Cloud #2649 (comment)
  • I can't remember why (forgot to write it down here) we considered it was a non problem.
• Why the new Apache can't have directory listing with today's (5th brownout)?
  • Because it does not have access to the JSON file (excluded from the htdocs, so it cannot generate a listing "on-the-go"
  • Also, the current (and only) rewrite condition triggering the redirection to the mirror is only checking for file existence in the Apache htdocs as per https://github.com/jenkins-infra/update-center2/blob/408b72d608f8151ab7c1e7735b1b0a0f11424e87/site/publish.sh#L177-L181
• Why Cloudflare does answer HTTP/404?
  • The mirrors ingress rule rewrites the requests to a directory by appending index.htmlto the request URI. It allows serving the existing index.html files (no try_files $uri/index.html $uri/ in R2). But it fails to serve the non existent index.html of course.

From now, we have the following choices:

• Keep the current behavior (e.g. HTTP/404 for requests targeting directory listing pages) and accept it is not a feature we want to keep
• Generates the directory listing on each mirror:
  • Dynamically (e.g. ala Apache) using Cloudlflare workers
    • Billing: https://developers.cloudflare.com/workers/platform/pricing/
    • Examples: https://developers.cloudflare.com/r2/examples/demo-worker/ and https://github.com/cmj2002/r2-dir-list?tab=readme-ov-file
  • Statically (e.g. in update_center2 job): most probably spinning up a Docker container, crawl the directories (15 to 20 requests) and add the index.html files in the dataset
  • VM based mirrors will be easy (Apache or Nginx would have directory listing OOB)
• Provides the files in the existing front Apache
  • Require to change the "fallback to mirror" rewrite rule. It is doable now: we have fixed the redirection priorities we had in the past weeks. Need to catch the exact URIs to send to mirrors
  • Would decrease the amount of requests (mostly error-ed requests) sent to cloudflare
  • Would increase the amount of requests served by our Apache. Let's keep in mind this content is not the initial "heavy bandwidth target" though.

[dduportal]

⚠️ We still have user-facing reported HTTP/404 on the "directory listing" pages (e.g. requests to a directory without an index.html). => not a blocker but annoying
...

Discussed during today's team meeting. We are proceeding with the last solution, following @daniel-beck 's recommendation.

E.g. we're updating the architecture to have Apache serving everything except the JSON files. We can do this now as we've solved the "Apache Redirections": it wasn't possible in the past months with the RedirectMatch directives.
=> it most probably will simplify the publish.sh process in update center2 job as well

[dduportal]

⚠️ We still have user-facing reported HTTP/404 on the "directory listing" pages (e.g. requests to a directory without an index.html). => not a blocker but annoying
...

Discussed during today's team meeting. We are proceeding with the last solution, following @daniel-beck 's recommendation.

E.g. we're updating the architecture to have Apache serving everything except the JSON files. We can do this now as we've solved the "Apache Redirections": it wasn't possible in the past months with the RedirectMatch directives. => it most probably will simplify the publish.sh process in update center2 job as well

• PR on the current UC: chore(wrappers - publish.sh) only send JSON files to mirrors update-center2#816 (ping @daniel-beck could i get your brain on this one? I don't mind scheduling a 1:1 meeting with you if it helps)
• Todo next (if it works as expected on a manual test):
  • Update crawler publication script to also publish to httpd (to have directoryt listing on /updates/)
  • Update mirrors.updates.jenkins.io ingress to stop rewriting directories to index.html

[dduportal]

Issues to track:

• Cloudflare access logs: [cloudflare] Collect access logs in a location where we can view and request them #4372
• HTTP/404 on directory listings: [Azure Update Center] Serve directory listing instead of redirecting to mirrors (ending in HTTP/404) #4373

[dduportal]

Update:

• Ready for a sixth brownout!
• Jenkins weekly and LTS cores are now setting up enforced HTTPS as per https://issues.jenkins.io/browse/JENKINS-73760. We should be able to get rid of the HTTP only stuff soon

[smerle33]

Update (preparing the 6th brownout):

• Missing /latest endpoint:
  • Tracked in sub-issue [Azure Update Center] HTTP/404 on directories which are symlinks #4375
  • Added this endpoint to the datadog monitor (so we'll detect this failure earlier when changing configurations): feat(synthetics_updatecenter) add /latest to monitoring datadog#268
• Announcing the 6th brownout:
  • Blog post: feat(blog) add a new post about the Update Center (6th brownout) jenkins.io#7658
  • Mailing lists (users, dev and infra): https://groups.google.com/g/jenkinsci-dev/c/maZUyqD7plg
  • status.jenkins.io: open brownout updatecenter 6 status#548
• DNS record change: feat(dns-records) open 6th Update Center brownout azure-net#312

[dduportal]

Update:

• 6th Brownout finished:
  • DNS revert: Revert "feat(dns-records) open 6th Update Center brownout" azure-net#313
  • status.jenkins.io: close 6th Azure UC brownout status#553
• Post Mortem: <incoming (wip)>