Debug Image Vulnerabilities

[fbdebugvuln]

Are there any plans to update the debug image with one that isn't filled with vulnerabilities?

See scan result here;
https://trivy.dev/results/?image=fluent/fluent-bit:1.9.9-debug

There are few guardrails set up in my place preventing me from pulling this image into my companies registry due to the vulnerabilities.

Also is this the debug Dockerfile https://github.com/fluent/fluent-bit/blob/master/dockerfiles/Dockerfile ?

[patrick-stephens]

The Dockerfile is multi-stage so yes there is a single definition for all architectures and image types.
The builds are all done via Github Actions here if you want to see how: https://github.com/fluent/fluent-bit/blob/master/.github/workflows/call-build-images.yaml

The debug image is not intended for production and has a large number of tools included specifically to help debug or develop (e.g. valgrind): it is very much intended to help debug things rather than be used directly.

Those vulnerabilities will likely be from the dependencies of the tools installed as the debug image is the production binaries (useful aspect of multi-stage builds) on top of a Debian image with Debian dependencies installed. There have been other requests for adding those tools in the debug image as this was required prior to the advent of debug containers in K8S for example.

I know it does not help your specific case which I think is to have a debug image available internally? What's the specific need for the debug image itself and maybe we can work from there?

[fbdebugvuln]

Thanks for all that info Patrick. Much appreciated.

My use case is quite simple on batch/cronjobs the k8s pod gets stuck in a NotReady state due to the fluent-bit side car is still running. I've been using an older version of the debug image to run a shell command to look for the existence of a file (that is created on the completion of the main container) and once found it will kill the fluent bit process.

k8s manifest looks something like this
main container

                - |
                  trap "touch /logs/terminated" EXIT
                  python ./hello_world.py

side car

                - |
                  TERMINATE_FILE=/logs/terminated
                  /fluent-bit/bin/fluent-bit -c /fluent-bit/etc/fluent-bit.conf &
                  PID=$!
                  while true
                    do
                      if [ -f "${TERMINATE_FILE}" ]
                      then
                        kill $PID
                        exit 0
                      fi
                    done

I understand this is more of a k8s issue than fluent-bit.

I tried following this tutorial https://suraj.io/post/how-to-gracefully-kill-kubernetes-jobs-with-a-sidecar/ to see if I can kill the fluent-bit side car by adding shareProcessNamespace and killing the process from the main container but it didn't work for me.

nobody@container[batch-job-xgjnc]:/app# ps -aux
USER        PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
nobody      1  0.0  0.0    956     4 ?        Ss   22:09   0:00 /pause
nobody      9  0.0  0.0  19756  2204 ?        Ss   22:09   0:00 /bin/sh -c sleep 9999 python /app/main.py
nobody     16  0.0  0.0  14160  1140 ?        S    22:09   0:00 sleep 9999
nobody     17  0.1  0.0  86560 30252 ?        Ssl  22:09   0:00 /fluent-bit/bin/fluent-bit /fluent-bit/bin/fluent-bit -c /fluent-bit/etc/fluent-bit.conf
nobody     28  0.0  0.0  19888  2744 ?        Ss   22:09   0:00 bash
nobody     40  0.0  0.0  53140  2612 ?        R+   22:10   0:00 ps -aux
nobody@container[batch-job-xgjnc]:/app# PID=${pgrep fluent}
nobody@container[batch-job-xgjnc]:/app# kill $PID
nobody@container[batch-job-xgjnc]:/app# pgrep sleep
16
nobody@container[batch-job-xgjnc]:/app# kill 16
nobody@container[batch-job-xgjnc]:/app# command terminated with exit code 137

$  kubectl get po | grep batch
batch-job-xgjnc                                  1/2     NotReady           1          2m35s

$  kubectl logs batch-job-xgjnc logging-sidecar
Fluent Bit v1.9.6
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io
[2022/10/13 22:21:14] [ info] [fluent bit] version=1.9.6, commit=b30141ba90, pid=16
[2022/10/13 22:21:14] [ info] [storage] version=1.2.0, type=memory-only, sync=normal, checksum=disabled, max_chunks_up=128
[2022/10/13 22:21:14] [ info] [cmetrics] version=0.3.5
[2022/10/13 22:21:14] [ info] [sp] stream processor started
[2022/10/13 22:21:14] [ info] [output:forward:forward.0] worker #0 started
[2022/10/13 22:21:14] [ info] [output:forward:forward.0] worker #1 started
[2022/10/13 22:21:38] [engine] caught signal (SIGTERM)
[2022/10/13 22:21:38] [ info] [input] pausing tail.0
[2022/10/13 22:21:38] [ warn] [engine] service will shutdown in max 5 seconds
[2022/10/13 22:21:39] [ info] [engine] service has stopped (0 pending tasks)
[2022/10/13 22:21:39] [ info] [output:forward:forward.0] thread worker #0 stopping...
[2022/10/13 22:21:39] [ info] [output:forward:forward.0] thread worker #0 stopped
[2022/10/13 22:21:39] [ info] [output:forward:forward.0] thread worker #1 stopping...
[2022/10/13 22:21:39] [ info] [output:forward:forward.0] thread worker #1 stopped

[patrick-stephens]

No worries, do you want to reach out on Slack - might be bit easier to discuss there? Or drop me an email to pat at calyptia.com.

Batch processing is something I've been advocating internally at Calyptia (it covers a myriad of sins including automated testing for example) so good to see more requests.

In your case it might be easier to use a custom container, e.g. with a Golang watcher to do this - doing it in a shell script is super flaky plus the debug images are humongous! There are a couple of good examples around this:

• Fluent Operator Image - this watches for the Fluent Bit config file to be updated then restarts the Fluent Bit process inside the container rather than redeploying the pod on config change: https://github.com/fluent/fluent-operator/blob/master/cmd/fluent-watcher/fluentbit/Dockerfile
• Couchbase Sidecar - I made this based on the above with some extra processing but I no longer work there: https://github.com/couchbase/couchbase-fluent-bit/

Note there are a few issues with batch processing currently: the main one being flushing - there is no guarantee when you exit that it will have flushed all data in the whole pipeline. I hit this issue in automated testing using exit_on_eof to read a file, process and output then exit when done but it occasionally exits too fast while records are still in flight so I had to revert to a timeout based approach of run for X time and then die. You could also monitor metrics output to see when output counters stop incrementing and kill then (you could do that with a golang prom client for example).

[fbdebugvuln]

Perfect thanks again Pat.

I was actually able to get the pod to go to a COMPELTE status after changing the script slightly to;

python -c "print('hello world')"
EXIT_CODE=$?
sleep 30s
echo "killing fluent bit"
pkill fluent
sleep 30s
if pgrep fluent >/dev/null 2>&1 
then
   echo "Fluent-bit still running, force killing"
   pkill -9 fluent
fi
sleep 15s
exit $EXIT_CODE

Job got marked as completed

$ kubectl get po | grep batch
batch-job-4744f                 0/2     Completed          0          115s

I had to install procps on my image to use the above command but I did some searching on my company's repository and found someone else was doing the same without installing that package. They take advantage of the /proc/[pid]/status file to find the fluent-bit PID and then kill it.

[patrick-stephens]

Glad you got it sorted, I would still say that debug image is far too big just for this though. You could use the UBI image we make for Red Hat certification as this has a shell and package manager so a bit smaller.

• https://github.com/calyptia/fluent-bit
• https://catalog.redhat.com/software/containers/calyptia/fluent-bit/61d8a3609ea53314a92f8e79

[fbdebugvuln]

I don't think I explained very well as what I was doing above was trying to kill the side car without the shell in the debug image. So the above testing was done with the none debug image.

What we currently run in production is have our main container (rhel-minimal) and our sidecar which has the fluent-bit debug image. To kill the side car we create a file (/logs/terminated) in the shared volume when the main container (rhel-minimal) ends and using the shell on the debug image we keep polling for this file and once it find it the command is issued to kill the side car.

Separately what I had been testing above was using the property shareProcessNamespace (see here for more info https://suraj.io/post/how-to-gracefully-kill-kubernetes-jobs-with-a-sidecar/) in our k8s manifest for the batch job. By enabling this I can see the side car processes from the main containers shell (rhel-minimal) which means I can kill the side car from the main container. So the shell scripts in my earlier posts are from the main container using the none debug image. Which means I no longer have to use the debug image.

Thanks again Pat for all your help on this.

[patrick-stephens]

Ah ok, great!

I'd be interested in a write-up on this btw, we've been looking at improving batch processing so having a documented example would be great. Let me know though if you do publish anything, we can always cross-promote as well.