Issues with Fluent Bit Monitoring Host System (Linux) Metrics in a Docker Container

[zRaphi]

Hello everyone,

I installed Fluent Bit as a Docker container to monitor the metrics of the host system (Linux System), not just the metrics of the container. I added /proc, /run/systemd, /sys, and / as volumes. However, it seems like the container is only monitoring itself.

There are also two errors in the logs:
[input:node_exporter_metrics:node_exporter_metrics.0] systemd init failed
[input:node_exporter_metrics:node_exporter_metrics.0] could not set systemd collector for Node Exporter Metrics plugin

Any Ideas?

[patrick-stephens]

Provide the full config and command you used to run it along with the versions, etc. used.

[zRaphi]

FluentBit Version 3.0.7

Docker-Compose Part:

  fluentbit:
    image: ${fluentbit_version}
    container_name: fluentbit
    logging:
      driver: "json-file"
      options:
        max-size: "5m"
        max-file: "2"
    command:
      - '--config=/fluent-bit/etc/fluent-bit.yml'
    privileged: true
    ports:
      - 2021
    networks:
       - observability
    volumes:
      - /proc:/host/proc:ro
      - /run/systemd:/run/systemd:ro
      - /sys:/host/sys:ro
      - /:/host/root:ro
      - /observability/fluent-bit.yml:/fluent-bit/etc/fluent-bit.yml:ro
      - /:/host:ro,rslave
    pid: host

fluent-bit.yml

service:
    flush: 1
    log_level: info
#    parsers_file: parsers.conf
pipeline:
    inputs:
        - name: node_exporter_metrics
          tag:  node_metrics
          scrape_interval: 2
    outputs:
        - name: prometheus_exporter
          match: node_metrics
          host: 0.0.0.0
          port: 2021
          # add user-defined labels
          add_label:
            - app fluent-bit
            - color blue
 #   inputs:
 #       - name: systemd
 #         tag:  host.*
 #         systemd_filter: _SYSTEMD_UNIT=docker.service
 #   outputs:
 #       - name: stdout
 #         match: "*"

I think that should be everything

[patrick-stephens]

Check the docs, it's all described there even with an example: https://docs.fluentbit.io/manual/pipeline/inputs/node-exporter-metrics#container-to-collect-host-metrics

[zRaphi]

But how do I integrate this into my Docker-Compose/Config-File construct? I have already tried specifying the paths in the input pipeline, but as far as I can see, this hasn't solved my problem. How can I tell if the data from the host system is being read correctly? And should the systemd error be resolved with this as well?

[zRaphi]

And i think it already uses the correct paths for procfs and sysfs, see log:

[2024/06/19 05:05:27] [ info] [input:node_exporter_metrics:node_exporter_metrics.0] path.procfs = /host/proc
[2024/06/19 05:05:27] [ info] [input:node_exporter_metrics:node_exporter_metrics.0] path.sysfs  = /host/sys

[patrick-stephens]

Yeah that's what I did and it worked fine.

[patrick-stephens]

The systemd error is nothing to do with this, that's probably down to incompatible systemd versions between host and container.

[1puls2is3]

In a completely different environment, I had the same problem.

My environment:
Debian 12.5 x86_64
kubernets 1.30.1
containerd v1.6.32
runc 1.1.12

I think this problem may be caused by a bug in node_exporter_metrics.

[agup006]

Is the problem the systemd metrics failing? Potentially you can turn off that collection and see where it repros?

[1puls2is3]

Thanks for your reply!
I don't know what “could not set systemd collector for Node Exporter Metrics plugin” means.
How can I turn off the collection of systemd metrics?
I see 3 systemd-related settings in the Node Exporter Metrics documentation:
https://docs.fluentbit.io/manual/pipeline/inputs/node-exporter-metrics
systemd_include_service_task_metrics = false ?

#9110
This page contains my earliest test configurations