Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Normandy CDN Migration #2341

Open
4 of 17 tasks
tiftran opened this issue Nov 23, 2022 · 4 comments
Open
4 of 17 tasks

Normandy CDN Migration #2341

tiftran opened this issue Nov 23, 2022 · 4 comments

Comments

@tiftran
Copy link
Contributor

tiftran commented Nov 23, 2022 •
edited
Loading

Migrate over to GCP

  • Use nginx/varnish to replicate the exact same responses we serve now, as an interim solution, as a diff dns first
  • Have QA do a sanity check by setting desktop to point to diff dns, run test suite for normandy
  • Use uptake telemetry to get healthy baseline of normandy calls
    Start with a 20% switch over and look at uptake telemetry after a day to ensure normandy calls are as expected
  • Switch over to original dns(100% if previous step is fine)
    Note: for the interest of time and cost and deciding it was somewhat low risk, we rolled out incrementally throughout a single day
  • Roll out 1.45.1 to prod which include caching changes to the v1 endpoint

Follow up work.....

  • Identify consumers of Normandy

    • Analyze user agents to understand the distribution of clients/versions/domains calling/being called
    • identify all the remote calls made by Normandy desktop client

  • Make the code changes for headers that we can get rid of

    • Run a dummy experiment to get baseline
    • Explore making server side changes to deprecate the dependency on certain headers ref:docs
    • Implement changes to remove headers from investigation (Normandy Server)
    • Implement changes (if any) to Normandy Client
    • Migrate CDN again to reflect code changes
    • Run a dummy experiment to ensure there are no breaking changes
    • Setup manual QA process to test clients explicitly hitting old and new CDN

  • Identity which endpoints are no longer used and which arent for deprecation plan
@jaredlockhart
Copy link
Collaborator

Uptake telemetry:

https://sql.telemetry.mozilla.org/queries/67658#219874

@jaredlockhart
Copy link
Collaborator

image

@tiftran
Copy link
Contributor Author

tiftran commented Dec 6, 2022

QA Jira: https://mozilla-hub.atlassian.net/browse/QA-1701?atlOrigin=eyJpIjoiNDBmODZlMzUzNmU5NDM1OWIzZDg5NGFhYWNhMWVlZDQiLCJwIjoiamlyYS1zbGFjay1pbnQifQ

@tiftran
Copy link
Contributor Author

tiftran commented Dec 13, 2022

Summary of what has happened so far:

Nov 29th: Normandy new CDN setup for Normandy was available in the stage environment

Nov 30th: tested 1%, 10%, 25% of requests were moved to the new CDN

Dec 1st: 100% of requests were moved to the new CDN

Dec 2nd: #2342 PR was merged

Dec 5th: v1.145.1 was tagged and pushed to stage, QA request was made

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jaredlockhart @tiftran