Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generate staging buildspec from Makefiles #841

Merged
merged 2 commits into from
Jun 3, 2022

Conversation

jaxesn
Copy link
Member

@jaxesn jaxesn commented May 20, 2022

Issue #, if available:

Description of changes:

The main purpose of this PR is to provide explicit dependency declaration to support auto generating the staging-bundle.yml.

The staging build, which uses that bundle, is a batch codebuild which builds "everything" in the build-tooling repo. The purpose of this is to have a process where everything is built from scratch, which is different than our dev build/release process which just builds the projects which changed.

In some cases we have the build triggers, in our dev build process, setup where if dependent projects change, it will also trigger the project which depend on it, but I suspect this is in complete. For the most this works out to be ok because we only have one version of the projects in this repo and dependencies are pulled using the GIT_TAG from repo. That means that if say distribution changes to a new version and harbor kicks off before the distribution job publishes the new tarball, with the new git_tag in the name, it will wait_for_tar to try and ensure it pulls the correct version. In the worst case this wait will timeout and the harbor build will fail, requiring a manual retrigger which would then use the newest version of the distribution dep.

The staging build/release building everything, using the batch build yml to model the dependencies, is to validate that everything is using the correct version and building in the proper order. Currently the staging-build.yaml has been manually created and maintain and has gotten out of date, for example there is no distribution dependency on the harbor build as their should be. The reason this ends up not being an issue is because the staging build builds to and pulls from the same folder/image tags as the dev build/release. I want to explore changing this to ensure the staging build is running from a "clean" env.

Before introducing the staging-build.yaml we used to use https://github.com/aws/eks-anywhere-build-tooling/blob/main/Makefile#L8 to model these dependencies since we would build everything from one code build job, use make release from the main Makefile. We do not do this anymore which makes this array of projects largely unnecessary and out of date.

We also had a psuedo-dependency model using the FETCH_BINARIES helper in makefiles which really just a way to standardize downloading the tars of other projects, there are really just a few examples of this across the repo.

The first commit in this PR changes FETCH_BINARIES into PROJECT_DEPENDENCIES with the main purpose of being able to use this information to regenerate the staging-bundle.yml ensuring it is always up to date and there is only one real source of truth for which projects depend on other projects.

This has been removed for now, but I may bring it back in another PR The second commit is more of a POC in how with deps modeled in the makefile structure, we could introduce an additional target to make it easier to build all the dependencies project locally if that is desired.

re: #801

/hold

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@eks-distro-bot eks-distro-bot added do-not-merge/work-in-progress do-not-merge/hold size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels May 20, 2022
@jaxesn jaxesn force-pushed the jgw/automated-buildspec branch 3 times, most recently from 7e5c2be to 6d02960 Compare June 3, 2022 13:17
@jaxesn jaxesn changed the title WIP: generate buildspec script Generate staging buildspec from Makefiles Jun 3, 2022
@jaxesn
Copy link
Member Author

jaxesn commented Jun 3, 2022

/unhold

@jaxesn
Copy link
Member Author

jaxesn commented Jun 3, 2022

/hold

Adding presubmit to verify this stays up to date

@ewollesen ewollesen self-requested a review June 3, 2022 15:39
Copy link
Contributor

@ewollesen ewollesen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shoot, these comments have been sitting here for a while, sorry.

build/lib/all_projects.sh Outdated Show resolved Hide resolved
build/lib/all_projects.sh Outdated Show resolved Hide resolved
@abhay-krishna
Copy link
Member

/retest

@jaxesn
Copy link
Member Author

jaxesn commented Jun 3, 2022

/unhold

Added the presubmit

@jaxesn jaxesn force-pushed the jgw/automated-buildspec branch 2 times, most recently from 3f2d8f6 to 9e6950d Compare June 3, 2022 21:07
@eks-distro-bot
Copy link
Collaborator

eks-distro-bot commented Jun 3, 2022

@jaxesn: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
tinkerbell-helm-presubmit 7e5c2be link true /test tinkerbell-helm-presubmit

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@abhay-krishna
Copy link
Member

/retest

@jaxesn
Copy link
Member Author

jaxesn commented Jun 3, 2022

/approve

@abhay-krishna
Copy link
Member

/lgtm
/approve

@eks-distro-bot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abhay-krishna, jaxesn

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [abhay-krishna,jaxesn]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@eks-distro-bot eks-distro-bot merged commit c11b830 into aws:main Jun 3, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved lgtm size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants