TOOMANYREQUESTS errors when downloading java-db from ghcr.io

[sakky016]

Question

Issue in java-db download
This is in follow-up to discussion this thread - https://github.com/orgs/community/discussions/139074#discussioncomment-10739491.

Shouldn't trivy check some sort of release-info/metadata before trying to download java-db? This release-info could indicate that a new java-db is available, perhaps by using content-hash of the available java-db). If new java-db is not available or has failed (just like this issue-https://github.com/aquasecurity/trivy-java-db/actions), then trivy should not attempt to download the artifact at all.

The current logic is dependent on the publishing process of java-db and the schedule of download is governed by NextUpdate field of metadata.json. With this approach trivy always attempts to download java-db (even if the one that is available is older or has failed). This logic does not consider that future publishing of java-db could fail.

Target

Container Image

Scanner

Vulnerability

Output Format

JSON

Mode

Standalone

Operating System

LInux

Version

No response

[knqyf263]

Yes, your suggestion makes sense. We already compare the content hash for misconfiguration checks to prevent the bundle from being downloaded unnecessarily. We can do the same for trivy-java-db as well.

trivy/pkg/policy/policy.go

Lines 172 to 175 in 37d549e

	digest, err := c.artifact.Digest(ctx)
	if err != nil {
	return false, xerrors.Errorf("digest error: %w", err)
	}

On the other hand, we have DownloadedAt to mitigate this kind of problem. For example, trivy-db checks DownloadedAt and will not download the database too frequently.

trivy/pkg/db/db.go

Lines 142 to 145 in 37d549e

	if now.Before(meta.DownloadedAt.Add(time.Hour)) {
	log.Debug("DB update was skipped because the local DB was downloaded during the last hour")
	return true
	}

I didn't find the same logic in trivy-java-db. Did we forgot to add it or am I missing something? @DmitriyLewen

trivy/pkg/javadb/client.go

Line 56 in 37d549e

if (meta.Version != SchemaVersion || meta.NextUpdate.Before(time.Now().UTC())) && !u.skip {

[DmitriyLewen]

@knqyf263 we have been implementing trivy-java-db at an accelerated pace (because maven servers work unstable) - so it looks like we really forgot to take this into account (although we are updating this field):

trivy/pkg/javadb/client.go

Line 77 in 8d0ae1f

meta.DownloadedAt = time.Now().UTC()

[knqyf263]

OK. Can we quickly implement it?

[DmitriyLewen]

I think so. We just need to add another if for DownloadedAt.
I will try to do that today.

But I think 1 hour (as for trivy-db) is too small.
What if we use 1 day?

[DmitriyLewen]

@knqyf263 #7592

[durcon]

BTW: What is the reason to load trivy-db every 6 hours?

2024-09-30T12:14:02Z	DEBUG	DB update was skipped because the local DB is the latest
2024-09-30T12:14:02Z	DEBUG	DB info	schema=2 updated_at=2024-09-30T06:14:28.338986152Z next_update=2024-09-30T12:14:28.338985871Z downloaded_at=2024-09-30T12:06:18.039677019Z

For Trivy 17.x I could read in the docs that it is downloaded every 12 hours. In the docs for Trivy 55 I don't find anything about it.

[DmitriyLewen]

Due to the current situation, perhaps we should think about reducing the update frequency of trivy-db
@knqyf263

[knqyf263]

@durcon It's written here.
https://aquasecurity.github.io/trivy/v0.55/docs/scanner/vulnerability/#vulnerability-database

@DmitriyLewen Users can specify --skip-db-update if they want to skip updates. They can create a cron workflow to pull the database and store it in the cache every 6 hours, 12 hours or 24 housrs, for example, and create another workflow to scan for vulnerabilities with --skip-db-update, like we did for test images.

[knqyf263]

You can use the cache. We plan to document it better.

[Ksoftcode]

I had the offline cache and CRON update every 12 hours implemented using Azure File Share and Jenkins for updates and CRON job. However, a new experience is that Trivy Java DB pulls from the docker hub every runtime instead of being cached like Trivy DB. Is there a reason why Trivy Java DB is treated differently?

[gnomefin]

For those who are still facing issues, according to maintainer's suggestion, this below snippet highlights the example quick fix code that I created:

• Using oras to fetch/pull the offline db
• store airgapped (offline DB) to cache
• caching condition
• The execution is always using --skip-db-update

(this is for trivy-db, if you are using java version, you can adjust it accordingly)

name: Trivy Scan
description: "This action is intended to run Trivy in an air-gapped environment."
inputs:
  trivy-image-ref:
    description: "Trivy image reference based on repository and tags"
    required: true
  trivy-gcp-artifact-registry:
    description: "Artifact registry in GCP"
    required: true
    default: me-central2-docker.pkg.dev

runs:
  using: composite
  steps:
    - name: GCloud Auth WIP Dev
      id: gcloud-auth
      uses: <org>/actions-workflows/actions/modules/workload-identity-provider/generic@main

    # Install Oras CLI to pull Trivy DB v2
    - name: Install Oras CLI
      shell: bash
      run: |
        curl -sSL https://github.com/oras-project/oras/releases/download/v0.16.0/oras_0.16.0_linux_amd64.tar.gz | tar -xz -C /usr/local/bin oras

    # Install Trivy
    - name: Install Trivy
      shell: bash
      run: |
        curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sh -s -- -b /usr/local/bin v0.55.2

    # Set up cache for Trivy database, include version in the cache key
    - name: Set Up Trivy Cache
      id: trivy-cache
      uses: actions/cache@v4
      with:
        path: ~/.cache/trivy/db
        key: ${{ runner.os }}-trivy-db-cache

    # Only download the Trivy offline database if cache miss
    - name: Download Trivy Offline Database
      if: steps.trivy-cache.outputs.cache-hit != 'true'
      shell: bash
      run: |
        oras pull ghcr.io/aquasecurity/trivy-db:2
        ls -lah

    # Only extract the database if cache miss
    - name: Set Up Trivy Database in Cache
      if: steps.trivy-cache.outputs.cache-hit != 'true'
      shell: bash
      run: |
        mkdir -p ~/.cache/trivy/db
        tar -xvf db.tar.gz -C ~/.cache/trivy/db

    # Run Trivy using the offline database
    - name: Run Trivy Vulnerability Scanner
      shell: bash
      run: |
        trivy --cache-dir ~/.cache/trivy image --exit-code 0 --format table --ignore-unfixed --severity CRITICAL,HIGH --skip-db-update ${{ inputs.trivy-image-ref }}

Still failed to fetch? You can wait until the rate limit period is over or we can use AWS ECR mirror to trigger fetch, then as soon as it is available to fetch, run it and store to cache.

[mebibite]

Getting the following error when setting this up with GitLab and using the Trivy Docker image:

2024-10-08T09:14:50Z ERROR [vulndb] The first run cannot skip downloading DB
2024-10-08T09:14:50Z FATAL Fatal error init error: DB error: database error: --skip-update cannot be specified on the first run

The cache files are present. Any ideas?

[DmitriyLewen]

Run Trivy in debug mode.
Trivy shows cache dir (e.g. 2024-10-08T15:38:42+06:00 DEBUG Cache dir dir="/Users/foo/Library/Caches/trivy")
and check that you correctly save cache with trivy-java-db.

[mebibite]

Thank you for the instant response, @DmitriyLewen!

Using the debug mode, I noticed that the application expects a 'db' subdirectory under the provided cache-dir. This matches the example above, I missed it. Creating the 'db' subdirectory and putting the files there fixed the issue.

[lebenitza]

Thank you @gnomefin for the workaround. I believe that caching should've already been a part of the official GitHub action: https://github.com/aquasecurity/trivy-action as it is proposed here: aquasecurity/trivy-action#399

Hopefully that gets merged soon.

[Ksoftcode]

Hello @Ksoftcode

We had some problems building trivy-java-db, but it should be fixed now.

Can you check the metadata.json file for trivy-java-db?

Hi Hi @DmitriyLewen, Thanks so much i got this resolved. I used caching method using Azure File Share and a Jenkins job that pulls once a day into the cache folder.
The cache is moved into directories /db for trivy db and /java-db for java index db. If anyone needs the script to get this up i can share. Many thanks

[mebibite]

This situation has really become unworkable for us. I will implement the workaround suggested by @gnomefin (thanks a lot for that), but I do hope that the developers take action to avoid this from happening for other users.

[MaxDNG]

Hi all, after updating to v0.56.1 we still have the issue when running from SemaphoreCI.
Here are the logs, they don't seem to be specific to java db though.

INFO	[vulndb] Need to update DB00:01
INFO	[vulndb] Downloading vulnerability DB...
INFO	[vulndb] Downloading artifact...	repo="ghcr.io/aquasecurity/trivy-db:2"
ERROR	[vulndb] Failed to download artifact	repo="ghcr.io/aquasecurity/trivy-db:2" err="OCI repository error: 1 error occurred:\n\t* GET https://ghcr.io/v2/aquasecurity/trivy-db/manifests/2: TOOMANYREQUESTS: retry-after: 802.099µs, allowed: 44000/minute"
FATAL	Fatal error	init error: DB error: failed to download vulnerability DB: OCI artifact error: failed to download vulnerability DB: failed to download artifact from any source

[MaxDNG]

See new discussion #7668