cmd: add new image-builder binary and initial `list-distros` #997

mvo5 · 2024-10-28T10:40:05Z

[draft as this is sill WIP and questions like where to get the authorative list of distros/repos and if we should embed them via "embed:" to have a simpler distribution mechanism etc]

This commit adds a new image-builder binary that will be capable
to build images standalone (daemonless). There are various use-cases
for this, the most important one is CI/CD pipelines.

This binary should eventually also replace the existing "cmd/list-images" and
"cmd/build" binaries (and potentially more) and it will cherry pick a lot of the
functionality from there. I decided against starting from one of the exiting binaries
because it allows a slightly cleaner start but this work is early so we could still
reconsider if people feel strongly it should be done the other way around.

thozza · 2024-11-04T11:39:29Z

cmd/image-builder/manifest.go

+	// XXX: what/how much do we expose here?
+	var options distro.ImageOptions


For on-prem, we use ImageOptions for the stuff that is passed on the CLI and is not part of the BP. E.g. image size.

This is something we need to think about. The image options, like Tomáš said, is (mostly) for things that come from the command line. This includes:

image size

ostree options (url, ref, and parent, when using composer-cli compose start-ostree).

https://github.com/osbuild/osbuild-composer/blob/a7cd521325504a43d3708f8f91ab63999862b279/internal/weldr/api.go#L2549-L2556

But it's also used for other things that get set automatically (or semi-automatically):

Facts: this shouldn't be exposed at all. It adds a value to the RHSM facts so we can see how the image was built when it registers back with subscription manager (hosted service vs on-prem). Perhaps we could add a new value for this now for the daemonless builder.

Subscription: Used only by the service (I think, https://github.com/osbuild/osbuild-composer/blob/a7cd521325504a43d3708f8f91ab63999862b279/internal/cloudapi/v2/compose.go#L1194). But might be useful to expose for on-prem builders.

PartitioningMode: This is a funny one. In osbuild-composer, it's set in the blueprint (https://github.com/osbuild/osbuild-composer/blob/a7cd521325504a43d3708f8f91ab63999862b279/internal/blueprint/customizations.go#L23), but it's passed down to images through the image options (https://github.com/osbuild/osbuild-composer/blob/a7cd521325504a43d3708f8f91ab63999862b279/internal/weldr/api.go#L2552). Not sure what we want to do there. Since it's in the BP in composer already, I think we should move it there in images too.

achilleas-k

I really like this PR (what it's starting, in particular). I realise there are many reasons why it's a draft, but it makes me worry that it will be forgotten if we don't move towards something more concrete and start merging soon.

I think we might get the ball rolling if we start small. A PR that only implements list-images for example would be very quick to merge, and after that we'd have the image-builder command ready to expand on and start using in testing (replacing the existing cmd/list-images).

achilleas-k · 2024-11-04T13:18:53Z

cmd/image-builder/main.go

+	// XXX: this will list 802 images right now, we need a sensible
+	// default here, maybe without --filter just list all available
+	// distro names?


I think it's fine to list everything if it's unfiltered. Perhaps in the human-readable version we can have a count on the last line with a suggestion to look into --filter, so it's clear that the user is getting too much output.
For the machine readable case, listing all is definitely ok as far as I'm concerned.

This commit creates a new `imagefilter` package that can be used to create a filter for the available images that can be build by the images library. Support for prefixes and fnmatch style globbing is available. This will eventually allows us to do: ``` $ image-builder --list-images --filter src:aarch64 --filter type:qcow* or $ image-builder --list-images --filter ami ``` The filtering can be both with or without a prefix, i.e. when doing a "simple" search all common fields (disro,arch,imageType) are checked but with a prefix search things can be narrowed down more. Initially the following prefixes filters are supported: "distro:" - the distro name, e.g. rhel-9, or fedora* "arch:" - the architecture, e.g. x86_64 "type": - the image type, e.g. ami, or qcow? "bootmode": - the bootmode, e.g. "legacy", "uefi", "hybrid" (Split out from the spike in osbuild#997)

This commit creates a new `imagefilter` package that can be used to create a filter for the available images that can be build by the images library. Support for prefixes and fnmatch style globbing is available. This will eventually allows us to do: ``` $ image-builder --list-images --filter arch:aarch64 --filter type:qcow* or $ image-builder --list-images --filter ami ``` The filtering can be both with or without a prefix, i.e. when doing a "simple" search all common fields (disro,arch,imageType) are checked but with a prefix search things can be narrowed down more. Initially the following prefixes filters are supported: "distro:" - the distro name, e.g. rhel-9, or fedora* "arch:" - the architecture, e.g. x86_64 "type": - the image type, e.g. ami, or qcow? "bootmode": - the bootmode, e.g. "legacy", "uefi", "hybrid" (Split out from the spike in osbuild#997)

This commit creates a new `imagefilter` package that can be used to create a filter for the available images that can be build by the images library. Support for prefixes and fnmatch style globbing is available. This will eventually allows us to do: ``` $ image-builder --list-images --filter src:aarch64 --filter type:qcow* or $ image-builder --list-images --filter ami ``` The filtering can be both with or without a prefix, i.e. when doing a "simple" search all common fields (disro,arch,imageType) are checked but with a prefix search things can be narrowed down more. Initially the following prefixes filters are supported: "distro:" - the distro name, e.g. rhel-9, or fedora* "arch:" - the architecture, e.g. x86_64 "type": - the image type, e.g. ami, or qcow? "bootmode": - the bootmode, e.g. "legacy", "uefi", "hybrid" (Split out from the spike in osbuild#997)

This commit creates a new `imagefilter` package that can be used to create a filter for the available images that can be build by the images library. Support for prefixes and fnmatch style globbing is available. This will eventually allows us to do: ``` $ image-builder --list-images --filter arch:aarch64 --filter type:qcow* or $ image-builder --list-images --filter ami ``` The filtering can be both with or without a prefix, i.e. when doing a "simple" search all common fields (disro,arch,imageType) are checked but with a prefix search things can be narrowed down more. Initially the following prefixes filters are supported: "distro:" - the distro name, e.g. rhel-9, or fedora* "arch:" - the architecture, e.g. x86_64 "type": - the image type, e.g. ami, or qcow? "bootmode": - the bootmode, e.g. "legacy", "uefi", "hybrid" (Split out from the spike in #997)

mvo5 · 2024-11-12T08:50:54Z

Just FTR - this will be transferred to https://github.com/osbuild/image-builder-cli (unless someone has concerns)

This commit contains an abreviated version of an osbuild jsonseq output log. Most of the repetitive data from the curl and rpm stages got removed as they don't add (much) to the understanding. The data was generated via: ``` $ sudo python3 -m osbuild --libdir . --monitor JSONSeqMonitor \ --export image --output-directory /tmp/out \ ./test/data/manifests/fedora-boot.json > osbuild-status-output.json ```

This commit adds support for the jsonseq based monitoring that osbuild provides. This is useful for machine readable status reporting.

This commit adds a `Status.Trace` string in the osbuildmonitor. Trace contains a single log line, usually very low-level or stage output bug useful for e.g. bug reporting. Should in general not be displayed to the user but the concatenation of all "trace" lines should give the same information as running osbuild on a terminal.

This commit adds `Status.Timestamp` so that it's easier to correlate what took how long when reconstructing the log (e.g. for bugreporting). See also: https://github.com/osbuild/images/compare/main...mvo5:osbuildmonitor-testdata?expand=1 for the full log.

This commit adds support cabilities to filter the list of distros based on a given search expression. Those are: ``` foo name:foo arch:i386 type:ami bootmode:uefi ``` and they will filter the list of distros.

This commit adds a new `image-builder` binary that will (eventually) be capable to build images standalone (daemonless). There are various use-cases for this, the most important one is CI/CD pipelines but also providing a common abstraction over the "images" library (and potentially later "otk"). This commit implements basic a `list-images` command with filtering and text and json output, e.g. ``` $ image-builder list-images --filter bootmode:none --filter name:fedora-39 --filter arch:s390x fedora-39 --arch s390x --type container $ ./image-builder list-images --filter bootmode:none --filter name:fedora-39 --filter arch:s390x --format=json {"distro":{"name":"fedora-39"},"arch":{"name":"s390x"},"image_type":{"name":"container"}} ```

…opy/paste friendly

This commit adds support to filter image based on (unresolved) packages. E.g.: ``` $ ./image-builder list-images --filter pkg:gdisk ... rhel-9.6 type:vhd arch:x86_64 ``` Based on an idea from Achilleas-k, many thanks.

This commit allows to change the filename when creating a manifest from an image type. This is useful when doing a UI for users, e.g. this will allow us to support: ``` $ image-builder build rhel-9 type:qcow2 --output foo.img ``` (this will also be useful for bootc-image-builder). Well, osbuild itself will still put it under a directory when doing main_cli.py:exports() but that is something orthogonal.

mvo5 requested a review from achilleas-k October 28, 2024 10:40

mvo5 force-pushed the add-image-builder-cmd branch 9 times, most recently from e727cd2 to e130485 Compare October 30, 2024 18:07

thozza reviewed Nov 4, 2024

View reviewed changes

achilleas-k reviewed Nov 4, 2024

View reviewed changes

mvo5 force-pushed the add-image-builder-cmd branch from 43fd9e4 to 2aebe55 Compare November 6, 2024 12:54

mvo5 mentioned this pull request Nov 6, 2024

pkg: add new imagefilter package with an ImageFilter #1015

Merged

mvo5 force-pushed the add-image-builder-cmd branch 2 times, most recently from eca4753 to 08ff6df Compare November 6, 2024 16:25

mvo5 mentioned this pull request Nov 7, 2024

imagefilter: add ResultFormatter type to support flexible output #1019

Merged

mvo5 force-pushed the add-image-builder-cmd branch from 08ff6df to 8c2dfa4 Compare November 9, 2024 18:18

mvo5 force-pushed the add-image-builder-cmd branch from 8c2dfa4 to 6890d8e Compare November 11, 2024 14:26

mvo5 force-pushed the add-image-builder-cmd branch 3 times, most recently from f6194f3 to b8cb4e8 Compare November 12, 2024 19:29

mvo5 force-pushed the add-image-builder-cmd branch 2 times, most recently from bd762f4 to f232368 Compare November 15, 2024 09:59

mvo5 and others added 19 commits November 15, 2024 16:00

osbuildmonitor: new package to support jsonseq monitoring from osbuild

6b062a1

This commit adds support for the jsonseq based monitoring that osbuild provides. This is useful for machine readable status reporting.

schutzfile: Update osbuild dependency commit ID

2af9d3a

ostree: skip MTLS on nil

ea197e1

disk: nicer formatting for NewPartitionTable docstring

cde7c4e

worker: parse ostree proxy early

3b039d6

image-builder: add new filters capbility to filter on distro

bf66aad

This commit adds support cabilities to filter the list of distros based on a given search expression. Those are: ``` foo name:foo arch:i386 type:ami bootmode:uefi ``` and they will filter the list of distros.

cli: implement messy/untested "image-builder manifest"

bb8f21e

cli: implement messy/untested "image-builder build"

2505333

fixup! move newRepoRegistry() into it's own file

49c4c05

fixup! rebase on distrofilter branch

06af986

fixup! allow "{distro,arch,type}:" prefix to build,manifest to make c…

1f9bb00

…opy/paste friendly

fixup! update to latest imagefilter-fmt branch

639bed7

imagefilter: add support to filter for pkg: prefixes

3ea8514

This commit adds support to filter image based on (unresolved) packages. E.g.: ``` $ ./image-builder list-images --filter pkg:gdisk ... rhel-9.6 type:vhd arch:x86_64 ``` Based on an idea from Achilleas-k, many thanks.

image-builder: add ib build --filename foo.img support

493d0cc

mvo5 force-pushed the add-image-builder-cmd branch 2 times, most recently from 313e543 to eba44b4 Compare November 15, 2024 16:16

wip: progress reporting via jsonseq-monitor

aaa2938

mvo5 force-pushed the add-image-builder-cmd branch from eba44b4 to aaa2938 Compare November 15, 2024 16:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd: add new image-builder binary and initial `list-distros` #997

cmd: add new image-builder binary and initial `list-distros` #997

mvo5 commented Oct 28, 2024

thozza Nov 4, 2024

achilleas-k Nov 4, 2024

achilleas-k left a comment

achilleas-k Nov 4, 2024

mvo5 commented Nov 12, 2024

		// XXX: what/how much do we expose here?
		var options distro.ImageOptions

cmd: add new image-builder binary and initial list-distros #997

Are you sure you want to change the base?

cmd: add new image-builder binary and initial list-distros #997

Conversation

mvo5 commented Oct 28, 2024

thozza Nov 4, 2024

Choose a reason for hiding this comment

achilleas-k Nov 4, 2024

Choose a reason for hiding this comment

achilleas-k left a comment

Choose a reason for hiding this comment

achilleas-k Nov 4, 2024

Choose a reason for hiding this comment

mvo5 commented Nov 12, 2024

cmd: add new image-builder binary and initial `list-distros` #997

cmd: add new image-builder binary and initial `list-distros` #997