Zstd(:chunked) work tracking checklist

[mtrmac]

• Zstd in general:
  • Switch to Zstd default, earlier than a Zstd:chunked default? Testing in DO NOT MERGE: Run tests with Zstd compression default podman#21571 ; Adjust some tests to work with Zstd in containers.conf podman#21793
  • Not a bug: pull of gzip + push of zstd changes image ID due to triggering a conversion to OCI: [rawhide+podman-next] podman push/pull registry roundtrip changes the image podman#20611
  • Pushes with Zstd compression prefer using schema1 manifests: Conversion to schema1 does not fail with Zstd layers, making it uncertain we correctly convert to OCI #2181 , Refuse compression to zstd when using schema1 #2196
  • Pushes with Zstd compression to schema-agnostic transports don’t trigger a conversion: Copies with Zstd compression to schema-agnostic transports don’t trigger schema conversion #2195 , Trigger a conversion to OCI when compressing to Zstd #2204 Zstd:chunked:
  • Blob reuse is not restricted by possible manifest schema options: Blob reuse decisions do not take into account manifest support #2205 , Make blob reuse choices manifest-format-sensitive, and allow conversions when writing to format-agnostic transports #2213
  • Uncompressed pushes of originally-compressed layers don’t correctly update MIME types (always a bug, but now breaks pushes of originally-Zstd to docker-archive: destinations): Copies of originally-compressed images from c/storage to uncompressed destinations don’t trigger MIME type updates #2182
• Zstd:chunked:
  • Exercise Podman tests with Zstd:chunked default: DO NOT MERGE: Run tests with Zstd:chunked compression default podman#21903
  • At a higher level, it’s unclear how to safely handle signed chunked images: what if the two versions don’t represent exactly the same filesystem?
  • copy.Options.EnsureCompressionVariantsExist would never stop adding zstd:chunked variants: copy.Options.EnsureCompressionVariantsExist doesn’t detect existing variants with zstd:chunked #2318
  • expectedLayerDiffIDFlag seems to use mismatching types: expectedLayerDiffIDFlag expectation mismatch #2602 , expectedLayerDiffIDFlag expectation mismatch #2602
  • Pulls on VFS fail outright: podman pull zstd:chunked-image, with vfs: not supported podman#24308 , Fall back from partial pull when on VFS storage#2140
  • Reusing already-present data can be optimized: Extend PutLayer to optimize reusing data from existing layers storage#1830, Update a comment after a c/storage update #2299, Propagate CompressedDigest/CompressedSize when reusing data from another layer #2583
  • See if the BlobCache needs updating for Zstd / Zstd:chunked: https://issues.redhat.com/browse/RUN-1124
  • Document zstd:chunked and encryption interaction: Document that zstd:chunked doesn’t make sense with encryption common#2117, Document that zstd:chunked is downgraded to zstd when encrypting common#2176, Document that zstd:chunked is downgraded to zstd when encrypting skopeo#2427, Document that zstd:chunked is downgraded to zstd when encrypting buildah#5759, Document that zstd:chunked is downgraded to zstd when encrypting podman#24113
  • Propose iterateTarSplit to upstream tar-split: Ensure chunked TOC and tar-split metadata are consistent storage#2035 (comment) . Filed Add tar/asm.IterateHeaders vbatts/tar-split#71 , Use tar-split/tar/asm.IterateHeaders now that it has been accepted storage#2116 .
  • blobPipelineDetectCompressionStep detects zstd:chunked as zstd, causing unnecessary recompression: around copy: do not fail if digest mismatches #1980 (comment) , Record zstd:chunked format, and annotations, in BlobInfoCache #2487 .
  • Pushes reusing chunked layers don’t create the required annotations, making the layers impossible to pull chunked: [WIP] compression: add specific prefix for zstd:chunked #2183 (review) / WIP HACK: Do not reuse zstd:chunked blobs #2185 , Record zstd:chunked format, and annotations, in BlobInfoCache #2487
  • zstd:chunked and layer encryption don’t make sense together: zstd:chunked and layer encryption don’t make sense together #2485
  • Blocker? TOC data and tar-split may be ambiguous: zstd:chunked metadata ambiguity storage#2014 , Ensure chunked TOC and tar-split metadata are consistent storage#2035
  • c/storage AdditionalLayerStore seems to be used for TOC-identified layers, incorrectly: Support additional layer store (patch for containers/storage) storage#795 (comment) , [Additional Layer Store] Use TOCDigest as ID of each layer (patch for c/storage) storage#1924
  • TarSplitChecksumKey not used in a layer ID: zstd:chunked blocker: TarSplitChecksumKey not used in a layer ID storage#1888 , Move the tar-split digest into the TOC storage#1902
  • blobPipelineCompressionStep would trigger a recompression of zstd:chunked if the user asks for zstd (except that we don’t currently detect zstd:chunked): around copy: do not fail if digest mismatches #1980 (comment) , Improve handling of zstd vs. zstd:chunked matching #2317
  • The c/storage “binary footer” code path does not work: zstd:chunked binary footer format is broken storage#1886
  • Outstanding items from chunked: generate tar-split as part of zstd:chunked storage#1627 (review) : Chunked cleanups storage#1844
  • Copies of chunked layers don’t mark the image as requiring Zstd support: Copies don’t set OCI1InstanceAnnotationCompressionZSTD on Zstd:chunked #2077 , Also annotate image instances using zstd:chunked as using zstd #2302
  • Chunked layers are visible in c/storage, and usable by other processes, while they are still being populated: Creation of Zstd:chunked layers seems racy #1979, store: new API ApplyStagedLayer  storage#1826 + storage: use the new ApplyStagedLayer interface #2301 .
  • PutBlobPartial with a non-chunked input and conversion in c/storage enabled doesn’t work: Fix c/storage destination with partial pulls #2288 (comment) , fixed there
  • Pushes of chunked layers fail because the UncompressedDigest fields stores a TOC digest, and layer digest validation fails: Fixed in copy: do not fail if digest mismatches #1980
  • Pulls of chunked layers do not reuse locally-existing layers: Part of copy: do not fail if digest mismatches #1980
  • In c/storage, we don’t sufficiently differentiate layers pulled by blob digest vs. TOC digest, and reuse is unclear: Part of copy: do not fail if digest mismatches #1980 , and chunked: generate tar-split as part of zstd:chunked storage#1627 (comment) . Fixed in Fix c/storage destination with partial pulls #2288 .
  • Pulls may reuse layers pulled by blob digests vs. TOC digests: Part of copy: do not fail if digest mismatches #1980 . Fixed in Fix c/storage destination with partial pulls #2288 .
  • Image deduplication contains a (required!) sanity check that TopLayer matches, but that might not be true if layers pulled by blob vs. TOC digests have different IDs: Part of copy: do not fail if digest mismatches #1980 . Fixed in Fix c/storage destination with partial pulls #2288 .
  • Reuse of local layers by TOC may trigger inconsistent metadata updates: Fix manifest updates when we match a layer by TOC digest #2294
  • A pull by TOC + conversion to non-OCI might parse data incorrectly: Cleanly fail when trying to obtain a DiffID of a non-OCI image #2295

[giuseppe]

@flouthoc PTAL

[rhatdan]

Should

Pushes reusing chunked layers don’t create the required annotations, making the layers impossible to pull chunked: https://github.com/containers/image/pull/2183#pullrequestreview-1725228215 / https://github.com/containers/image/pull/2185

Be closed?

[cgwalters]

For the record I think it was me that recently added a zstd:chunked label on issues - there's some more stuff there. In particular it seems like containers/storage#1928 is somewhat close to a blocker for wide scale deployment.