Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Filelog receiver log mount points #1332

Open
paraniraja opened this issue Sep 6, 2024 · 2 comments
Open

Filelog receiver log mount points #1332

paraniraja opened this issue Sep 6, 2024 · 2 comments
Labels
chart:operator Issue related to opentelemetry-operator helm chart

Comments

@paraniraja
Copy link

Hi,

Using opentelmetry-operator, how to mount /var/log/pods to daemonset?

Tried with version "0.68.1", only conf file is mounted.

Thanks

Copy link

dosubot bot commented Sep 6, 2024

Hey @paraniraja, I'm here to help you with your coding conundrums! Let's squash those bugs together! 🤖

To mount /var/log/pods to a DaemonSet using the OpenTelemetry operator, you need to modify the DaemonSet configuration to include the necessary volume and volumeMounts. Here is an example configuration:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: opentelemetry-collector
  namespace: default
spec:
  selector:
    matchLabels:
      name: opentelemetry-collector
  template:
    metadata:
      labels:
        name: opentelemetry-collector
    spec:
      containers:
        - name: opentelemetry-collector
          image: otel/opentelemetry-collector:0.68.1
          volumeMounts:
            - name: varlogpods
              mountPath: /var/log/pods
              readOnly: true
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
      volumes:
        - name: varlogpods
          hostPath:
            path: /var/log/pods
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers

This configuration mounts the /var/log/pods directory from the host to the /var/log/pods directory in the container, allowing the OpenTelemetry collector to access the logs.

Additionally, you may want to configure the Filelog Receiver to collect logs from these mounted directories. Here is an example configuration for the Filelog Receiver:

receivers:
  filelog:
    include:
      - /var/log/pods/*/*/*.log
    exclude:
      - /var/log/pods/*/otel-collector/*.log
    start_at: beginning
    include_file_path: true
    include_file_name: false
    operators:
      - type: router
        id: get-format
        routes:
          - output: parser-docker
            expr: 'body matches "^\\{"'
          - output: parser-crio
            expr: 'body matches "^[^ Z]+ "'
          - output: parser-containerd
            expr: 'body matches "^[^ Z]+Z"'
      - type: regex_parser
        id: parser-crio
        regex: '^(?P<time>[^ Z]+) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) ?(?P<log>.*)$'
        output: extract_metadata_from_filepath
        timestamp:
          parse_from: attributes.time
          layout_type: gotime
          layout: '2006-01-02T15:04:05.999999999Z07:00'
      - type: regex_parser
        id: parser-containerd
        regex: '^(?P<time>[^ ^Z]+Z) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) ?(?P<log>.*)$'
        output: extract_metadata_from_filepath
        timestamp:
          parse_from: attributes.time
          layout: '%Y-%m-%dT%H:%M:%S.%LZ'
      - type: json_parser
        id: parser-docker
        output: extract_metadata_from_filepath
        timestamp:
          parse_from: attributes.time
          layout: '%Y-%m-%dT%H:%M:%S.%LZ'
      - type: move
        from: attributes.log
        to: body
      - type: regex_parser
        id: extract_metadata_from_filepath
        regex: '^.*\/(?P<namespace>[^_]+)_(?P<pod_name>[^_]+)_(?P<uid>[a-f0-9\-]{36})\/(?P<container_name>[^\._]+)\/(?P<restart_count>\d+)\.log$'
        parse_from: attributes["log.file.path"]
        cache:
          size: 128
      - type: move
        from: attributes.stream
        to: attributes["log.iostream"]
      - type: move
        from: attributes.container_name
        to: resource["k8s.container.name"]
      - type: move
        from: attributes.namespace
        to: resource["k8s.namespace.name"]
      - type: move
        from: attributes.pod_name
        to: resource["k8s.pod.name"]
      - type: move
        from: attributes.restart_count
        to: resource["k8s.container.restart_count"]
      - type: move
        from: attributes.uid
        to: resource["k8s.pod.uid"]

This configuration ensures that the Filelog Receiver collects logs from the specified directories and processes them appropriately [1][2].

To continue talking to Dosu, mention @dosu.

@dosubot dosubot bot added the chart:operator Issue related to opentelemetry-operator helm chart label Sep 6, 2024
@vc43
Copy link

vc43 commented Oct 10, 2024

@paraniraja Have you found a solution?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
chart:operator Issue related to opentelemetry-operator helm chart
Projects
None yet
Development

No branches or pull requests

2 participants