Garbage collector not clean redis keys

[nevlkv]

Expected behavior and actual behavior:
Garbage collector not cleanup redis keys, and failed with message

2023-09-06T02:37:23Z [INFO] [/jobservice/job/impl/gc/garbage_collection.go:414]: 2937 blobs and 812 manifests are actually deleted 2023-09-06T02:37:23Z [INFO] [/jobservice/job/impl/gc/garbage_collection.go:415]: The GC job actual frees up 54343 MB space. 2023-09-06T02:37:23Z [ERROR] [/jobservice/job/impl/gc/garbage_collection.go:445]: failed to clean registry cache error retrieving 'blobs::*' keys: WRONGPASS invalid username-password pair or user is disabled., pattern blobs::*

Steps to reproduce the problem:

Starts registry cleanup

Versions:

• harbor version: 2.8.4

Additional context:

external redis with default user

_REDIS_URL_REG: redis://:password@harbor-redis-master:6379/2?idle_timeout_seconds=30

[wy65701436]

the error points the auth info is incorrect, can you check that at your end?

[nevlkv]

password correct

redis-cli -n 2 -a <password>
127.0.0.1:6379[2]> info keyspace
# Keyspace
db0:keys=133200,expires=133085,avg_ttl=2212096
db1:keys=2036,expires=1748,avg_ttl=83530815
db2:keys=5784,expires=156,avg_ttl=85886397
db5:keys=63,expires=54,avg_ttl=2360250

[nevlkv]

garbage service used _REDIS_URL_REG: ?

[MinerYang]

Hi @nevlkv

• Is this a fresh install harbor? Have you changed the redis password after installation?
• Could you also check/provide other redis url, for example _REDIS_URL_CORE?

[nevlkv]

1. current instance was upgraded from 2.7.1, redis is external and not changed
2. we use default redis account and password always identical

kubectl get cm harbor-core -n=io  -o yaml
apiVersion: v1
data:
  _REDIS_URL_CORE: redis://:<password>@harbor-redis-master:6379/0?idle_timeout_seconds=30
  _REDIS_URL_REG: redis://:<password>@harbor-redis-master:6379/2?idle_timeout_seconds=30

i can auth with via redis-cli and flushdb

GC use same connection string

[viceice]

Seeing same error:

023-09-17T00:02:59Z [ERROR] [/jobservice/job/impl/gc/garbage_collection.go:515]: failed to clean registry cache failed to scan keys: WRONGPASS invalid username-password pair or user is disabled., pattern blobs::*

I've no password for my external redis, i'm using network policies to limit redis access to harbor.

_REDIS_URL_CORE: redis://keydb:6379/0?idle_timeout_seconds=30
_REDIS_URL_REG: redis://keydb:6379/2?idle_timeout_seconds=30

Harbor version: v2.9.0

[viceice]

It seems the redis url is not passed to the job service via env.

I only see this inside config.yml if job service:

worker_pool:
  workers: 10
  backend: "redis"
  redis_pool:
    redis_url: "redis://keydb:6379/1"
    namespace: "harbor_job_service_namespace"
    idle_timeout_second: 3600

[viceice]

We probably need to set ?

_REDIS_URL_REG: "{{ template "harbor.redis.urlForRegistry" . }}"

harbor-helm/templates/jobservice/jobservice-cm-env.yaml

Line 15 in 86dd759

JOBSERVICE_WEBHOOK_JOB_HTTP_CLIENT_TIMEOUT: "{{ .Values.jobservice.notification.webhook_job_http_client_timeout }}"

[nevlkv]

error stable

2023-09-21T07:55:57Z [ERROR] [/jobservice/runner/redis.go:123]: Job 'GARBAGE_COLLECTION:c09d285cf0794f361744acab' exit with error: run error: error retrieving 'blobs::*' keys: WRONGPASS invalid username-password pair or user is disabled.

[nevlkv]

Current state

2023-09-24T02:31:20Z [INFO] [/jobservice/job/impl/gc/garbage_collection.go:414]: 3177 blobs and 562 manifests are actually deleted
2023-09-24T02:31:20Z [INFO] [/jobservice/job/impl/gc/garbage_collection.go:415]: The GC job actual frees up 37506 MB space.
2023-09-24T02:31:20Z [ERROR] [/jobservice/job/impl/gc/garbage_collection.go:445]: failed to clean registry cache error retrieving 'blobs::*' keys: WRONGPASS invalid username-password pair or user is disabled., pattern blobs::*

[MinerYang]

Hi @nevlkv ,
Could you try to monitor the registry log when pushing images, if there's any error msg?

[nevlkv]

Sometimes on pull 404
with "blob unknown"

time="2023-09-26T11:19:16.21575386Z" level=error msg="response completed with error" auth.user.name=harbor err.code="blob unknown" err.detail=sha256:7dbc1adf280e1aa588c033eaa746aa6db327ee16be705740f81741f5e6945c86 err.message="blob unknown to registry" go.version=go1.20.7 http.request.host=REGISTRY http.request.id=cdc0816e-2548-40d7-b33c-3f2f9a26837b http.request.method=HEAD http.request.remoteaddr=*.*.*.* http.request.uri="/v2/REPO/blobs/sha256:7dbc1adf280e1aa588c033eaa746aa6db327ee16be705740f81741f5e6945c86" http.request.useragent="docker/20.10.22 go/go1.18.9 git-commit/42c8b31 kernel/5.15.0-83-generic os/linux arch/amd64 UpstreamClient(docker-compose/1.29.2 docker-py/6.1.3 Linux/5.15.0-83-generic)" http.response.contenttype="application/json; charset=utf-8" http.response.duration=104.591881ms http.response.status=404 http.response.written=157 vars.digest="sha256:7dbc1adf280e1aa588c033eaa746aa6db327ee16be705740f81741f5e6945c86" vars.name="REPO"

[nevlkv]

If default user in URI undefined rise error

$ redis-cli -u redis://:<password>@harbor-redis-master:6379/2?idle_timeout_seconds=30 info keyspace
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
AUTH failed: WRONGPASS invalid username-password pair or user is disabled.
$ redis-cli -u redis://default:<password>@harbor-redis-master:6379/2?idle_timeout_seconds=30 info keyspace 
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
# Keyspace
db0:keys=1732,expires=1732,avg_ttl=1826800
db1:keys=116,expires=65,avg_ttl=44559991
db2:keys=1,expires=0,avg_ttl=0
db5:keys=4,expires=0,avg_ttl=0

define user in values

   type: external
   external:
     addr: "harbor-redis-master:6379"
+    username: "default"

and

kubectl get cm harbor-core -n=io  -o yaml
apiVersion: v1
data:
  _REDIS_URL_CORE: redis://:<password>@harbor-redis-master:6379/0?idle_timeout_seconds=30
  _REDIS_URL_REG: redis://:<password>@harbor-redis-master:6379/2?idle_timeout_seconds=30

became

kubectl get cm harbor-core -n=io  -o yaml
apiVersion: v1
data:
  _REDIS_URL_CORE: redis://default:<password>@harbor-redis-master:6379/0?idle_timeout_seconds=30
  _REDIS_URL_REG: redis://default:<password>@harbor-redis-master:6379/2?idle_timeout_seconds=30

and GC success !!!

[viceice]

Interestingly a manual run works, but the sheduled runs are always failing. 😕

[nevlkv]

same error after upgrade to 2.10

scheduled fail, manual success

upgraded 08/02/2024

[github-actions]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[viceice]

not stale 😞

[nevlkv]

successful only manual job

[nevlkv]

execute success only manual

[nevlkv]

If GC scheduled and settings of harbor changed after, job became failed:

registry=# select * from schedule where callback_func_name='GARBAGE_COLLECTION'
registry-# ;
  id  |       creation_time        |        update_time         |    vendor_type     | vendor_id |    cron     | callback_func_name |                                                                                                         callback_func_param                                                                                                          | cron_type |       extra_attrs        |  revision  
------+----------------------------+----------------------------+--------------------+-----------+-------------+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+--------------------------+------------
 1009 | 2022-07-21 10:09:37.790778 | 2022-07-21 10:09:37.790778 | GARBAGE_COLLECTION |        -1 | 0 0 0 * * * | GARBAGE_COLLECTION | {"trigger":null,"deleteuntagged":true,"dryrun":false,"extra_attrs":{"delete_untagged":true,"dry_run":false,"redis_url_reg":"redis://redis:<any_password>@harbor-redis-master:6379/2?idle_timeout_seconds=30","time_window":2}} | Daily     | {"delete_untagged":true} | 1717372800
(1 row)

redis_url_reg static and not changes!

[github-actions]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[wy65701436]

@nevlkv can you share your settings when installing your harbor? You are seeing the error only happen at the scheduled GC, right?

[twhiteman]

I've run into this issue as well (upgraded Harbor from 2.6.3 to 2.10.1) where scheduled GC runs fail to clear the redis cache due to an auth issue.

I noticed in the harbor execution table, that the redis_url_reg field is different between a manual and scheduled run:

> select * FROM execution WHERE vendor_type = 'GARBAGE_COLLECTION';
-[ RECORD 18 ]-+
id             | 8106990
vendor_type    | GARBAGE_COLLECTION
vendor_id      | -1
status         | Error
status_message |
trigger        | SCHEDULE
extra_attrs    | {"delete_untagged":false,"dry_run":false,"freed_space":412868683,"purged_blobs":1352,"purged_manifests":668,"redis_url_reg":"redis+sentinel://172.16.1.1:26379,172.16.1.2:26379,172.16.1.3:26379/redismaster/2?idle_timeout_seconds=30","time_window":2,"workers":0}
start_time     | 2024-08-27 00:02:49.632401
end_time       | 2024-08-27 00:05:30
revision       | 4
update_time    | 2024-08-27 00:06:05

-[ RECORD 19 ]-+
id             | 8113128
vendor_type    | GARBAGE_COLLECTION
vendor_id      | -1
status         | Success
status_message |
trigger        | MANUAL
extra_attrs    | {"delete_untagged":false,"dry_run":false,"freed_space":29574525,"purged_blobs":1021,"purged_manifests":507,"redis_url_reg":"redis+sentinel://USERNAME:[email protected]:26379,172.16.1.2:26379,172.16.1.3:26379/redismaster/1?idle_timeout_seconds=30","time_window":2,"workers":0}
start_time     | 2024-08-27 17:06:27.378128
end_time       | 2024-08-27 17:09:10
revision       | 4
update_time    | 2024-08-27 17:09:22

Failed: "redis+sentinel://172.19.6.10:26379,172.19.6.30:26379,172.19.6.50:26379/redismaster/2?idle_timeout_seconds=30"
Success: "redis+sentinel://USERNAME:[email protected]:26379,172.19.6.30:26379,172.19.6.50:26379/redismaster/1?idle_timeout_seconds=30"

Note the successful one has a username|password AND there is different database reference (redismaster/1|2).

Also note that I have not changed the Harbor redis configuration since the initial installation.