Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Image repository port pruned from reflection request after 1000+ images #343

Open
springroll12 opened this issue Feb 16, 2023 · 3 comments
Open

Image repository port pruned from reflection request after 1000+ images #343

springroll12 opened this issue Feb 16, 2023 · 3 comments
Labels
blocked/upstream Blocked by an upstream dependency or issue bug Something isn't working

Comments

@springroll12
Copy link

Hello all,

I'm experiencing an issue where an ImageRepository fails to find the latest tags when there are more than 1000 image tags in the repo. We run our docker repository over https on port 5000 (not 443/80). What seems to happen is when the image reflector requests the latest tags with the n=1000 tag, the request does not contain the port.

All image repositories which have less than 1000 tags do not cause an error, so I presume still include the port.

We are running Nexus3 OSS repo manager (v 3.22) in case it makes a difference.

Image repository:

---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageRepository
metadata:
  name: not-working-image-repository
  namespace: flux-system
spec:
  image: <repo>:5000/<not-working-image>
  interval: 1m0s
  secretRef:
    name: docker-registry-credentials-image-automator

Note that working image repos seem to differ in ONLY the image name.

Error:

{
  "level": "error",
  "ts": "2023-02-16T14:30:05.582Z",
  "msg": "Reconciler error",
  "controller": "imagerepository",
  "controllerGroup": "image.toolkit.fluxcd.io",
  "controllerKind": "ImageRepository",
  "ImageRepository": {
    "name": "not-working-image-repository",
    "namespace": "flux-system"
  },
  "namespace": "flux-system",
  "name": "not-working-image-repository",
  "reconcileID": "8c492911-547a-4077-b438-fae9ec0f3b71",
  "error": "GET https://<repo>/v2/<not-working-image>/tags/list?last=not-working-tag-b1ee0122-462089352&n=1000: unexpected status code 404 Not Found: \n<!DOCTYPE html>\n<html lang=\"en\">\n<head>\n  <title>404 - Nexus Repository Manager</title><SNIP.. 404 html..>",
  "stacktrace": "sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235"
}

Image repository which works:

curl "https://<repo>:5000/v2/<working-image>/tags/list" | jq '.tags | length'
391

Image repository which does not work:

curl "https://<repo>:5000/v2/<not-working-image>/tags/list" | jq '.tags | length'
1039

Great work on flux! We use it every day at work and it has been pretty solid for the last year!
@hiddeco
Copy link
Member

hiddeco commented Feb 16, 2023

Going through our code, this is likely an upstream bug in e.g. github.com/google/go-containerregistry as we do not do much more than calling

image-reflector-controller/controllers/imagerepository_controller.go

Line 358 in ff67fcd

tags, err := remote.List(ref.Context(), options...)

While I do think this should be supported (and fixed), I think it is worth mentioning that solutions like GHCR do not allow querying over 1K tags in general.

@hiddeco hiddeco added bug Something isn't working blocked/upstream Blocked by an upstream dependency or issue labels Feb 16, 2023
@hiddeco
Copy link
Member

hiddeco commented Feb 16, 2023 •
edited
Loading

Thinking more about this, have you checked if Nexus3 OSS by any chance does not just provide a wrong "next" URL in the JSON output and/or response headers if you limit to 1000? I remember issues with Helm repository indexes with such solutions (Artifactory IIRC) which used to (or are still advertising) "wrong" URLs when a port was configured.

@springroll12
Copy link
Author

This is a good point. We are running Nexus3 behind a reverse proxy, which handles the 5000 port bits. When I limit the number of tags returned to 100 or so I can see the Link header being sent with the repo url minus the port.

< Link: <https://<repo>/v2/<not-working-image>/tags/list?n=100&last=<last-tag>>; rel="next"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
blocked/upstream Blocked by an upstream dependency or issue bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@hiddeco @springroll12