Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

in_opentelemetry: add tag_from_uri support to http2 codepath #8963

Merged

Conversation

nuclearpidgeon
Copy link
Contributor

@nuclearpidgeon nuclearpidgeon commented Jun 15, 2024

For some reason the new HTTP/2 server codepaths in the OpenTelemetry input plugin don't have any support for the tag_from_uri plugin option.

This PR adds the same v1_metrics / v1_logs / v1_traces tags that the tag_from_uri behaviour in the previous flb_downstream-based non-HTTP/2 codepath.

Strictly speaking, this is arguably not fully correct, as the "URI"s in the gRPC cases are more complex:

  • opentelemetry.proto.collector.metrics.v1.MetricsService/Export
  • opentelemetry.proto.collector.traces.v1.TracesService/Export
  • opentelemetry.proto.collector.logs.v1.LogsService/Export

However it's probably a good enough first step given that anyone trying to filter by telemetry signal type would already be using these v1_metrics / v1_logs / v1_traces tags, and it seems overly complex to have totally different longer tags just for gRPC input.

As I said in an earlier comment on #8734, the "proper" fix for this IMO is to switch to a tag_from_signaltype option or something like that. But this is more work and might count as a breaking config change. This PR at least brings the same behaviour as the non-HTTP/2 config case to the HTTP/2 config case.


"Next-step" fix for #8734 (previous PR #8881 only addressed cases where http2 plugin option was set to false).


Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
service:
    log_level: debug
pipeline:
    inputs:
        - name: opentelemetry
          tag_from_uri: true
          http2: true
    outputs:
        - name: 'stdout'
          match: 'v1_logs'

cURL command for testing:

curl --header "Content-Type: application/json" --request POST --data '{"resourceLogs":[{"resource":{},"scopeLogs":[{"scope":{},"logRecords":[{"timeUnixNano":"1660296023390371588","body":{"stringValue":"{\"message\":\"dummy\"}"},"traceId":"","spanId":""}]}]}]}'   http://localhost:4318/v1/logs
  • Debug log output from testing the change
[2024/07/01 10:20:54] [ info] [fluent bit] version=3.1.0, commit=5aab3eb02c, pid=16393
[2024/07/01 10:20:54] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2024/07/01 10:20:54] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/07/01 10:20:54] [ info] [cmetrics] version=0.9.1
[2024/07/01 10:20:54] [ info] [ctraces ] version=0.5.1
[2024/07/01 10:20:54] [ info] [input:opentelemetry:opentelemetry.0] initializing
[2024/07/01 10:20:54] [ info] [input:opentelemetry:opentelemetry.0] storage_strategy='memory' (memory only)
[2024/07/01 10:20:54] [debug] [opentelemetry:opentelemetry.0] created event channels: read=21 write=22
[2024/07/01 10:20:54] [debug] [downstream] listening on 0.0.0.0:4318
[2024/07/01 10:20:54] [ info] [input:opentelemetry:opentelemetry.0] listening on 0.0.0.0:4318
[2024/07/01 10:20:54] [debug] [stdout:stdout.0] created event channels: read=24 write=25
[2024/07/01 10:20:54] [ info] [sp] stream processor started
[2024/07/01 10:20:54] [ info] [output:stdout:stdout.0] worker #0 started
[2024/07/01 10:21:05] [debug] [input:opentelemetry:opentelemetry.0] attributes missing
[2024/07/01 10:21:05] [debug] [task] created task=0x7dec24023470 id=0 OK
[2024/07/01 10:21:05] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[0] v1_logs: [[1660296023.1698112429, {}], {"log"=>"{"message":"dummy"}"}]
[2024/07/01 10:21:05] [debug] [out flush] cb_destroy coro_id=0
[2024/07/01 10:21:05] [debug] [task] destroy task=0x7dec24023470 (task_id=0)
  • Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • Run local packaging test showing all targets (including any new ones) build.
  • Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • Documentation required for this feature

Backporting

  • Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

@nuclearpidgeon nuclearpidgeon force-pushed the opentelemetry-tagfromuri-http2fix branch from e0ae915 to 66ae903 Compare June 15, 2024 13:09
@nuclearpidgeon nuclearpidgeon force-pushed the opentelemetry-tagfromuri-http2fix branch from 66ae903 to 5aab3eb Compare July 1, 2024 08:23
@nuclearpidgeon nuclearpidgeon changed the title add tag_from_uri support to OpenTelemetry input's new http2 codepath in_opentelemetry: add tag_from_uri support to http2 codepath Jul 1, 2024
@nuclearpidgeon
Copy link
Contributor Author

@edsiper @cosmo0920 I've fixed up the formatting of the commit messages / PR titles on this one. It should properly fix the issue in #8734

tags are still v1_logs/v1_metrics/v1_traces even in gRPC cases

Signed-off-by: Stewart Webb <[email protected]>
@nuclearpidgeon
Copy link
Contributor Author

@edsiper back to you to hopefully get this merged

@lecaros
Copy link
Contributor

lecaros commented Sep 27, 2024

@edsiper LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants