Image policy not selecting the latest prerelease version properly

[poojaac]

We have configured the ECR repository to be the ImageRepository with ecr-auto-login enabled and image policy as below to select the latest pre-release versions. But the policy is not picking up the latest image pushed to the ECR

image-repository.yaml

apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageRepository
metadata:
  name: context-worker-dev
  namespace: pftp-da-team
spec:
  image: 446102611596.dkr.ecr.eu-central-1.amazonaws.com/ecs/pftp/da/dev/context-worker
  interval: 1m0s

image-policy.yaml

apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImagePolicy
metadata:
  name: context-worker-dev
  namespace: pftp-da-team
spec:
  imageRepositoryRef:
    name: context-worker-dev
  policy:
    semver:
      range: ~3.3.x-0

Attached the list of images avaiable in ECR

Image Policy is picking the version 446102611596.dkr.ecr.eu-central-1.amazonaws.com/ecs/pftp/da/dev/context-worker:3.3.0-SNAPSHOT.feature.CS-9971-fix-sonar-issues.4 which is much older than other images

Same issue is caused with other image policies as well where latest images are not picked always, but sometime it picks the latest one

[darkowlzz]

Hi, I think it's doing the right thing, comparing the pre-release values lexically in ASCII sort order as described in https://semver.org/. In this case, comparing the selected version with all the other tags that were created later in terms of date, the selected one seems to be the greatest in terms of semver pre-release comparison.
We use https://github.com/masterminds/semver library for comparing the versions.

[darkowlzz]

I just did a web search and found this website that helps you compare semvers https://semvercompare.azurewebsites.net/ . It's not using the same semver library, but the results should be almost the same. You can use it to verify if the ImagePolicy is selecting the correct version.

[poojaac]

Yes the image policy is properly selecting the version as per the semver comparison website. But since we want the policy to pick up the latest pre-release version pushed to the ECR. How can we tackle this?

There are no much examples on how to select the latest pre-release versions. So it would be helpful if you can provide some logic for this

[darkowlzz]

There's a whole docs page about it https://fluxcd.io/docs/guides/sortable-image-tags/ 🙂

[poojaac]

We cant use both semver and numerical options together in the policy. If we dont specify semver range to pick the pre-release version then the filter will not pickup the pre-release versions

spec:
  imageRepositoryRef:
    name: archive-worker-dev
  filterTags:
    pattern: '^3.3.0.*(?P<ts>[0-9])$'
    extract: '$ts'
  policy:
    # semver:
    #   range: ^3.3.x-0
    numerical:
      order: asc

[relu]

Hello @poojaac! Try something like this:

spec:
  imageRepositoryRef:
    name: archive-worker-dev
  filterTags:
    pattern: '^(?P<ver>3\.3\.[0-9]+).*\.(?P<inc>[0-9]+)$'
    extract: '$ver-$inc'
  policy:
    semver:
      range: ^3.3.x-0

This will extract the relevant version parts into a semver compatible format witha a sortable pre-release part.

[poojaac]

Hi @relu Thanks for the hint. I am able to see image-policy picking the latest image by build time. However, I am seeing the weird issue where sometimes image-policy will not identify the latest image in the registry and will be picking the old images from the registry. I tried deleting the image-policy from the cluster and created again with same semver version range as before, that time it was able able to identify and pickup the latest image from registry. How could we address this issue since deleting the policy and recreating would not always help us in automated process.

[darkowlzz]

ImagePolicy reevaluates the tags whenever ImageRepository updates. In order to debug this, you can check the status of ImageRepository if it has the latest tags. It'll only scan for new tags at the set spec.interval. The ImageRepository status contains a count of the tags it found and scan time. You can use these information to check if ImageRepository has updated or not.

For example:

status:
  canonicalImageName: ghcr.io/stefanprodan/podinfo
  conditions:
  - lastTransitionTime: "2022-08-24T20:39:05Z"
    message: successful scan, found 34 tags
    reason: ReconciliationSucceeded
    status: "True"
    type: Ready
  lastScanResult:
    scanTime: "2022-08-24T20:39:05Z"
    tagCount: 34
  observedGeneration: 1

Or just use kubectl to get the precise information:

$ kubectl get imagerepositories.image.toolkit.fluxcd.io podinfo
NAME      LAST SCAN              TAGS
podinfo   2022-08-24T20:39:05Z   34

Flux CLI can also be useful:

$ flux get images repository podinfo
NAME    LAST SCAN                       SUSPENDED       READY   MESSAGE
podinfo 2022-08-25T02:11:06+05:30       False           True    successful scan, found 34 tags

Also, while debugging, instead of deleting and creating the ImagePolicy again, you can try manually triggering a reconciliation of ImageRepository that'll also trigger ImagePolicy to update by using the flux CLI reconcile command:

$ flux reconcile image repository podinfo
► annotating ImageRepository podinfo in flux-system namespace
✔ ImageRepository annotated
◎ waiting for ImageRepository reconciliation
✔ scan fetched 34 tags

[poojaac]

Thanks. I will try reconciling image repository if the image-policy is not picking the latest image next time and let you know if that helps.