Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(sources): fix unexpected timestamp field #21558

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

rwakulszowa
Copy link

When log namespacing is enabled, exec and http_server sources include a redundant timestamp field when decoding is enabled. When decoding is not enabled, they put the content under a "message" key, instead of directly in the root object.

In the case of exec, the issue was caused by passing a hardcoded LogNamespace value to the decoder. Fixed by passing the parsed value.

In the case of http_server, the decoder ignored the global LogNamespace. Fixed by passing the correctly parsed value to the decoder.

Fixes #20404.

When log namespacing is enabled, `exec` and `http_server` sources include
a redundant `timestamp` field when decoding is enabled. When decoding
is not enabled, they put the content under a _"message"_ key, instead
of directly in the root object.

In the case of `exec`, the issue was caused by passing a hardcoded
`LogNamespace` value to the decoder. Fixed by passing the parsed value.

In the case of `http_server`, the decoder ignored the global
`LogNamespace`. Fixed by passing the correctly parsed value to the
decoder.

Fixes vectordotdev#20404.
@rwakulszowa rwakulszowa requested a review from a team as a code owner October 19, 2024 21:18
@bits-bot
Copy link

bits-bot commented Oct 19, 2024

CLA assistant check
All committers have signed the CLA.

@github-actions github-actions bot added the domain: sources Anything related to the Vector's sources label Oct 19, 2024
@rwakulszowa
Copy link
Author

Tested manually with the following config (copied verbatim from #20404):

schema:
  log_namespace: true

sources:
  source_exec:
    type: exec
    command: ['echo', '{"log": "123"}']
    mode: scheduled
  source_http_server:
    type: http_server
    address: 0.0.0.0:80
  source_socket:
    type: socket
    address: 0.0.0.0:9000
    mode: tcp

sinks:
  sink_console:
    inputs:
      - source_*
    type: console
    encoding:
      codec: json

Please double check, though. I'm not a ddog / vector expert, so it won't hurt to be extra careful here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
domain: sources Anything related to the Vector's sources
Projects
None yet
Development

Successfully merging this pull request may close these issues.

exec and http sources add a .timestamp field to the event after decoding
2 participants