Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid "creating repository" step in bulk if there is nothing new for it. #1179

Open
1 task done
mirror176 opened this issue Oct 13, 2024 · 0 comments
Open
1 task done

Comments

@mirror176
Copy link

Prerequisites

  • Have you checked for an existing issue describing your idea?

What is your proposal?

Skip creating repository step if there were no new/updated/removed packages to process.

What is the existing behavior, if any?

Creating repository step currently is performed when specifying a rebuild of 1+ ports even if there was no new packages added to the list, no base or ports tree updates, no changed options, and no packages had to be rebuilt or removed other than failures.

What is the motivation / use case for the change?

Performance optimization: package repository rebuilding can take a significant amount of time+resources as the total repository (not just what was requested for building) grows larger and depending on the CPU, RAM, and disk performance.

Did you consider any alternatives?

If it is for a single port rebuild attempt, test can be used in place of bulk to avoid the rebuild.

Is this really a ports feature request?

no

Example

An example run on poudriere-devel-3.4.99.20240811 and older+slower hardware where almost 8 minutes was reported as used to update a repository (usually contains just under 3k packages) that should not have been altered. The point of reruning it was to rebuild with different options to try to track if one causes/resolves the failure but no options were changed since the last bulk run across all packaged items.


# poudriere bulk -J2 -j local -p local sysutils/vobcopy
[00:00:00] Creating the reference jail... done
[00:00:36] Mounting system devices for local-local
[00:00:36] Stashing existing package repository
[00:00:36] Mounting ccache from: /var/cache/ccache
[00:00:36] Mounting ports from: /usr/ports/
[00:00:36] Mounting packages from: /usr/local/poudriere/data/packages/local-local
[00:00:36] Mounting distfiles from: /usr/ports/distfiles
[00:00:36] Copying /var/db/ports from: /usr/local/etc/poudriere.d/local-options
[00:00:36] Appending to make.conf: /usr/local/etc/poudriere.d/make.conf
/etc/resolv.conf -> /usr/local/poudriere/data/.m/local-local/ref/etc/resolv.conf
[00:00:36] Starting jail local-local
Updating /var/run/os-release done.
[00:00:36] Will build as nobody:nobody (65534:65534)
[00:00:37] Copying host static ccache from /usr/local/bin/ccache
[00:00:38] Ports supports: FLAVORS SUBPACKAGES SELECTED_OPTIONS
[00:00:38] Inspecting /usr/local/poudriere/data/.m/local-local/ref//usr/ports for modifications to git checkout... no
[00:04:06] Ports top-level git hash: 597c6c40b912 
[00:04:06] Acquiring build logs lock for local-local... done
[00:04:07] Logs: /usr/local/poudriere/data/logs/bulk/local-local/2024-10-12_17h21m27s
[00:04:07] WWW: http://127.0.0.1/poudriere//build.html?mastername=local-local&build=2024-10-12_17h21m27s
[00:04:07] Loading MOVED for /usr/local/poudriere/data/.m/local-local/ref/usr/ports
[00:04:07] Gathering ports metadata
[00:04:07] Calculating ports order and dependencies
[00:04:07] Sanity checking the repository
[00:04:07] Trimming IGNORED and blacklisted ports
[00:04:07] Checking packages for incremental rebuild needs
[00:05:18] Deleting stale symlinks... done
[00:05:18] Deleting empty directories... done
[00:05:18] Unqueueing existing packages
[00:05:18] Sanity checking build queue
[local-local] [2024-10-12_17h21m27s] [pkgqueue_sanity_check] Queued: 2  Built: 0  Failed: 0  Skipped: 0  Ignored: 0  Fetched: 0  Tobuild: 2   Time: 00:01:12
[00:05:18] Recording filesystem state for prepkg... done
[00:05:18] Processing PRIORITY_BOOST
[00:05:18] Building 2 packages using up to 2 builders
[00:05:18] Hit CTRL+t at any time to see build progress and stats
[00:05:18] [01] [00:00:00] Builder starting
[00:05:18] [02] [00:00:00] Builder starting
[00:05:19] [01] [00:00:01] Builder started
[00:05:19] [02] [00:00:01] Builder started
[00:05:19] [01] [00:00:00] Inspecting print/texinfo | texinfo-7.1_6,1: determining shlib requirements
[00:05:19] [02] [00:00:00] Building   sysutils/vobcopy | vobcopy-1.2.1
[00:05:21] [02] [00:00:02] Finished   sysutils/vobcopy | vobcopy-1.2.1: Failed: configure
[00:05:21] Stopping 2 builders
[00:05:21] Creating pkg repository
Creating repository in /tmp/packages: 100%
Packing files for repository: 100%
[00:13:17] Committing packages to repository: /usr/local/poudriere/data/packages/local-local/.real_1728779684 via .latest symlink
[00:13:17] Removing old packages
[00:13:17] Failed ports: sysutils/vobcopy:configure
[00:13:17] Ignored ports: print/texinfo
[local-local] [2024-10-12_17h21m27s] [committing] Queued: 2  Built: 0  Failed: 1  Skipped: 0  Ignored: 1  Fetched: 0  Tobuild: 0   Time: 00:09:11
[00:13:17] Logs: /usr/local/poudriere/data/logs/bulk/local-local/2024-10-12_17h21m27s
[00:13:17] WWW: http://127.0.0.1/poudriere//build.html?mastername=local-local&build=2024-10-12_17h21m27s
[00:13:17] Cleaning up
[00:13:17] Unmounting file systems

Additional context

Currently this issue may be usable as a workaround for #420 to get a repo from packages but that should still be its own separate thing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant