One FILTER to one OUTPUT and another FILTER to another output

[vijaynauti]

Hi All

I have configured Fluent-bit for my application deployed on ECS Fargate where i am sending my logs to CloudWatch log and OpenSearch, I have some ECS Metadata attached with my logs like container name and container_ID, these details i need to pass only with OpenSearch but not with CloudWatch log.

Is there is any way to customize this.

Below is my custom configuration.

[INPUT]
Name forward
unix_path /var/run/fluent.sock
Tag test_tag_cw
Mem_Buf_Limit 100MB
[INPUT]
Name forward
unix_path /var/run/fluent.sock
Tag test_tag_es
Mem_Buf_Limit 100MB
[FILTER]
Name record_modifier
Match test_tag_cw
Record app-version applog
Remove_key container_id
[FILTER]
Name record_modifier
Match test_tag_es
Record app-version applog
Remove_key container_name
[OUTPUT]
Name cloudwatch
Match test_tag_cw
region us-east-1
log_group_name fluent-bit-cloudwatch-v1
log_stream_prefix From-fluent-bit-
auto_create_group true
[OUTPUT]
Name OpenSearch
Match test_tag_es
Host ${my_DNS}
Port 443
Index search-logging-v1
Type _doc
AWS_Auth On
AWS_Region us-east-1
tls On
Suppress_Type_Name On
retry_limit 3

[vijaynauti]

@PettitWesley

[PettitWesley]

Does the log_key option for cloudwatch_logs not achieve what you want? https://github.com/aws-samples/amazon-ecs-firelens-examples/tree/mainline/examples/fluent-bit/cloudwatchlogs#what-if-i-just-want-the-raw-log-line-from-the-container-to-appear-in-cloudwatch

[INPUT]
Name forward
unix_path /var/run/fluent.sock
Mem_Buf_Limit 100MB

FireLens will auto-add this input. Please do not add it yourself. This will lead to issues if not removed.

• https://github.com/aws/aws-for-fluent-bit/blob/mainline/troubleshooting/debugging.md#tutorial-replicate-an-ecs-firelens-task-setup-locally
• https://aws.amazon.com/blogs/containers/how-to-set-fluentd-and-fluent-bit-input-parameters-in-firelens/

[vijaynauti]

@PettitWesley then Mem_Buf_Limit option if i want to customize, Where should i control this in above case

[PettitWesley]

For FireLens, you need to follow:

1. https://github.com/aws-samples/amazon-ecs-firelens-examples/tree/mainline/examples/fluent-bit/oomkill-prevention#2-mem_buf_limit
2. https://aws.amazon.com/blogs/containers/how-to-set-fluentd-and-fluent-bit-input-parameters-in-firelens/

[vijaynauti]

@PettitWesley

Below is the custom configuration i am using right now..... But issue is while sending logs to CloudWatch i don't want Container_id and container_name, Where while sending logs to OpenSearch i want these parameter to persist there. How should i resolve this.

[FILTER]
Name record_modifier
Match ss-firelens*
Remove_key container_id
Remove_key container_name
Remove_key ecs_cluster
Remove_key ecs_task_arn
Remove_key ecs_task_definition
Remove_key source
[OUTPUT]
Name cloudwatch
Match ss-firelens*
region us-east-1
log_group_name fluent-bit-cloudwatch-v1
log_stream_prefix From-fluent-bit-
auto_create_group true
[OUTPUT]
Name OpenSearch
Match **
Host *******************
Port 443
Index sstest
Type _doc
AWS_Auth On
AWS_Region us-east-1
tls On
Suppress_Type_Name On

[PettitWesley]

Can you please try this: https://github.com/aws-samples/amazon-ecs-firelens-examples/tree/mainline/examples/fluent-bit/cloudwatchlogs#what-if-i-just-want-the-raw-log-line-from-the-container-to-appear-in-cloudwatch

And then remove your custom record_modifier filter. Opensearch will get all metadata. Cloudwatch will just get the value of the log_key.