YAML config format with multiple pipelines

[alex-rowe]

We want to provide a default fluent-bit config and then provide the option for additional configs to be loaded as necessary.

For example, we always want to log file1.log and file2.log, and then leave it up to someone else to add/include configs for file3-5.log as needed.

This feels like we should use multiple pipeline: configurations, and include them in the top level of the YAML config, but:

1. Can we include multiple pipelines?
2. What is the effect of doing so? do they get concatenated into a single config at runtime anyway or are they self contained?
3. If they're combined/separated, what do we need to do regarding tag/match? for example, will match: * get applied to all of them, or only the ones inside that pipeline.

I was looking for a config flag or something to dump the in memory configuration to see what it ends up as (taking into accounts includes, pipelines etc), but couldn't find anything.

[allewell]

I'm doing something similar to your requirement - having an include block in the global config that specifies individual yaml files (with pipeline at the top of each). Each pipeline has an input and filters block, and I'm tagging each unique input and then matching on the filters as to only operate on the relevant logs.

It helps to imagine that everything gets virtually constructed as a single yaml file. So if you do something like match:* in an individual pipeline file, it could affect other pipelines that you've included.

For those reasons, I have a single "common" pipeline included at the bottom of the include block that does global cleanup (like, match:* remove log ), and an output block. Order seems to be respected here.

My only desire is to have the parsers defined within each pipeline yaml file. But according to the docs they still need to be in a global parsers.conf file :(