Image tags delayed to be stored for the GHCR and GCR Pull Through Cache Repositories - Replication does not work

[FelixRodriguezJara]

Expected behavior and actual behavior:
When a pull is done over a pull-through cache repository using GHCR and GCR, the image should be stored in the cache together with their tags. However, the image is cached, but no tags are in place. After some time, the tags are available, but, at that time, replication has gone through and it hasn't taken any effect.

Steps to reproduce the problem:
docker login harbor
docker pull harbor-host/ghcr/image-name
docker pull harbor-host/gcr/image-name

Versions:

• harbor version: Harbor 2.11.1
• docker engine version: N/A
• docker-compose version: N/A

Additional context:

• Harbor config files:
  I have set up the GHCR and GCR Pull Through Cache registries via terraform using the following manifest:

resource "harbor_registry" "gcr" {
  provider_name = "docker-registry"
  name          = "gcr"
  endpoint_url  = "https://gcr.io"
}

resource "harbor_registry" "ghcr" {
  provider_name = "github"
  name          = "ghcr"
  endpoint_url  = "https://ghcr.io"
}

Then, I've got two projects liked to those registries:

resource "harbor_project" "ghcr_mirror" {
  name                        = "ghcr"
  public                      = false
  vulnerability_scanning      = true
  enable_content_trust        = false
  enable_content_trust_cosign = false
  registry_id                 = harbor_registry.ghcr/gcr.registry_id
}

• Log files:
  When I tried usign provider_name="google-gcr" and "github-gcr", harbor-core logs were having errors due to invalid adapter, that is why I ended up setting those up like that.

Thank you very much!

[stonezdj]

However, the image is cached, but no tags are in place because containerd send a HEAD request to get the digest of the tag, and then send Pull request to pull image by digest, it is possible that only the digest is cached, but the tag is not guaranteed to be created.

[FelixRodriguezJara]

Thank you very much for your answer, @stonezdj! So, if that is the case, wouldn't it be a problem for the event based replication? If tags do not exist, the images do not get replicated. That is our concern as we have a global Harbor and then replications to the Harbor instance in China.

[stonezdj]

Could you please describe your requirements in detail?
The event based replication is not a reliable solution, it could fail with many factors, especially the source project is a proxy cache project. but I checked the webhook, even without tag created, the webhook even fired even without tag.

{
  "type": "PUSH_ARTIFACT",
  "occur_at": 1728959120,
  "operator": "harbor#proxy-cache-service",
  "event_data": {
    "resources": [
      {
        "digest": "sha256:22f27168517de1f58dae0ad51eacf1527e7e7ccc47512d3946f56bdbe913f564",
        "tag": "sha256:22f27168517de1f58dae0ad51eacf1527e7e7ccc47512d3946f56bdbe913f564",
        "resource_url": "harbor.local/dockerhub_proxy/library/busybox@sha256:22f27168517de1f58dae0ad51eacf1527e7e7ccc47512d3946f56bdbe913f564"
      }
    ],
    "repository": {
      "date_created": 1728959120,
      "name": "library/busybox",
      "namespace": "dockerhub_proxy",
      "repo_full_name": "dockerhub_proxy/library/busybox",
      "repo_type": "private"
    }
  }
}

[FelixRodriguezJara]

@stonezdj , thank you very much for looking into this issue. We have two Harbor registries, one acting as a global registry and another one region specific. We want to replicate everything to the region specific Harbor registry, including the assets stored in the pull-through cache repository. That's why we rely on the event based replication. To work around the issue with the tags, we have scheduled a replication over the pull-through cache projects. Anyway, if the tags are not present the scheduled replication doesn't work either. What is the recommendation for this scenario?

Thank you very much.