Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: support git-cache-rs [backport 2024.01] #20321

Merged

Conversation

MrKevinWeiss
Copy link
Contributor

Backport of #20311

Contribution description

This PR adds support for the Rust version of git-cache.

The shell version that we've integrated in dist/tools/git is quite a mess, barely maintainable. But most importantly, it is slow, and gets slower over time, due to the way the cache stores tags. Also, it doesn't support sparse checkouts. Crude measurements show that for our CI builds, for cortex-m builds 20-50% of time spent on each built are spent on checking out "pkg/cmsis".

git-cache-rs' syntax is compatible with regular git clone to the point where any git clone ... command that doesn't work with just changing to git cache clone is considered a bug.
Unfortunately that requires some changes. This PR introduces git-cache-rs support as fully optional, it'll only be used if GIT_CACHE_RS is set to the binary name. The legacy built-in shell git-cache should continue working like before.

git-cache-rs works a bit differently than the shell git-cache: instead of having one large bare cache repository that adds repos as remote, git-cache-rs just stores a "--mirror" repository for each cached repo. This is much simpler, although it might use a bit more space.

git-cache-rs supports sparse checkouts, which has been implemented here for pkg/cmsis. I've updated dwq and the murdock container, and am currently testing those changes on ci-staging.riot-os.org.

Testing procedure

I've been testing this PR together with the CI support in RIOT-OS/riotdocker#240 for a day or so on https://ci-staging.riot-os.org/.

Issues/PRs references

@MrKevinWeiss MrKevinWeiss added Area: build system Area: Build system Area: pkg Area: External package ports CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR Process: release backport Integration Process: The PR is a release backport of a change previously provided to master labels Jan 31, 2024
@riot-ci
Copy link

riot-ci commented Jan 31, 2024

Murdock results

✔️ PASSED

a658c0a pkg: ensure "dynamic sparse paths" are up-to-date

Success Failures Total Runtime
8629 0 8629 08m:33s

Artifacts

Copy link
Contributor

@kaspar030 kaspar030 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK.

@kaspar030
Copy link
Contributor

Is this waiting for bors!?

@kaspar030 kaspar030 added this pull request to the merge queue Jan 31, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to no response for status checks Jan 31, 2024
@kaspar030 kaspar030 added this pull request to the merge queue Jan 31, 2024
Merged via the queue into RIOT-OS:2024.01-branch with commit 8fff061 Jan 31, 2024
30 checks passed
@MrKevinWeiss MrKevinWeiss deleted the backport/2024.01/git-cache-rs branch February 1, 2024 09:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area: build system Area: Build system Area: pkg Area: External package ports CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR Process: release backport Integration Process: The PR is a release backport of a change previously provided to master
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants