Structured Logging For Fluent Bit's own logs

[PettitWesley]

Currently, Fluent Bit logs are just strings printed like this:

[2022/06/30 06:09:22] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log stream 152.stream.tail.0.ip-10-1-128-166.us-east-2.compute.internal.default.myapp-5468c5d4d7-n2swr.myapp.fleet in log group group.tail.0.ip-10-1-128-166.us-east-2.compute.internal.default.myapp-5468c5d4d7-n2swr.myapp
[2022/06/30 06:09:22] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Created log stream 152.stream.tail.0.ip-10-1-128-166.us-east-2.compute.internal.default.myapp-5468c5d4d7-n2swr.myapp.fleet
[2022/06/30 06:09:22] [ warn] [record accessor] translation failed, root key=metadata
[2022/06/30 06:09:22] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log stream 154.stream.tail.0.ip-10-1-12u8-166.us-east-2.compute.internal.default.myapp-5468c5d4d7-n2swr.myapp.toy in log group group.tail.0.ip-10-1-128-166.us-east-2.compute.internal.default.myapp-5468c5d4d7-n2swr.myapp

However, it is often hard for users to make sense of these logs. Fluent Bit is itself a logging tool and so it should have an ideal experience for its own logs.

The best practice these days is structured logging where the logs are JSON events. Each log should have lots of context added to it. For example, users often complain that a single chunk or request failure leads to lots of error messages and it is hard to figure out which errors are all for the same failure. This gets worse with workers, since with multithreading means that multiple threads are all outputting logs which get mashed together into a single stream.

An ideal experience would be something like the user can enable structured logging:

[SERVICE]
    structured_logging On

And the logs would be JSON events with useful metadata:

{
    "msg": "Creating log stream 154.stream.tail.0.ip-10-1-12u8-166.us-east-2.compute.internal.default.myapp-5468c5d4d7-n2swr.myapp.toy in log group group.tail.0.ip-10-1-128-166.us-east"
    "plugin": "output:cloudwatch_logs"
    "chunk": "flb.2332423"
    "request": "23423492304293"
}

Basically, we add the chunk ID to all logs so that the user can search for a chunk ID and see all logs for it. And then also we introduce the concept of Fluent Bit randomly generating its own request IDs so that all failures/errors that are part of a single request can be associated together.

This makes it easy to run queries on your Fluent Bit logs. If you search for a single request ID, you see all error logs for that request, whereas today, you could see a series of error messages from different components and it wouldn't be clear that all were part of the same request.

[agup006]

I'm a fan of this, I also think @pwhelan might have been looking at this at some point as well

[patrick-stephens]

Structured logging with the option to add custom metadata would be a great addition I think. Particularly for plugins as you can then add a lot more specific context.

Not all of Fluent Bit's log statements currently follow the same regex either so it can be hard to completely parse them.