Fluentibit is writing in my nfs share and cost me too much

[ashraf133]

Bug Report

Describe the bug
Fluentbit installed on ec2 with its own EBS volume and i have /share/logs mount in AWs EFS
all fluentbit conf file and lock and conf and /var/log/message file are in the EBS
all monitored logs are in /share/logs
Fluentbit is reading logs from an AWS EFS share and should write nothing inside but in aws bill i see that cost is doubling when fluentbit is up and the cause is write actions on EFS

To Reproduce

• put fluentbit in ec2 with ebs and put monitored logs in EFS

• Steps to reproduce the problem:

Expected behavior
Increase only read iops but i see increase of both read and write

• Configuration:
• EC2 redhat 8
• fluentbit 3.0.7
• AWS EFS

[edsiper]

@ashraf133 is this the default one provided in AWS or a custom one you have added ?

would you mind share your config ? I would love to learn more how is being used. cc: @agup006

[ashraf133]

Hello,

i installed official fluentbit 3.0.7 version on ec2 redhat 8.
all log files are in /var/log/efs which is a mounted fs
all fluentbit conf is in /etc/fluentbit which is local ebs disk
all fluentbit logs are in /var/log/messages which is local ebs disk
all buffered data are in /tmp/flb-storage and fluent-bit in ebs local disk also

cat /etc/fluent-bit/fluent-bit.conf

@INCLUDE conf/*.conf
@INCLUDE filter.conf

[SERVICE]
    # Flush
    flush        150
    daemon       Off
    # Log_Level
    log_level    info
    # Parsers File
    parsers_file parsers.conf

    # Plugins File
    plugins_file plugins.conf

    # HTTP Server
    http_server  Off
    http_listen  0.0.0.0
    http_port    2020
    storage.path /tmp/flb-storage/
    storage.backlog.mem_limit 500M
    storage.type filesystem

here is an example of monitored file :

cat /etc/fluent-bit/conf/XXXX1.log.conf

[INPUT]
    name tail
    path /var/log/efs/XXXX1.log
    parser json

    Path_Key    filename
    DB          /etc/fluent-bit/lock/XXXX1.lock
    DB.locking  true
    Tag 11
    Buffer_Chunk_Size 64K
    Buffer_Max_Size 1M
    Mem_Buf_Limit 5M
    Offset_Key offset
    storage.type filesystem

[OUTPUT]
    Name s3
    Match 11
    bucket mybucket
    region eu-west-3
    compression gzip
    store_dir_limit_size 500M
    retry_limit 10
    total_file_size 50M
    send_content_md5 true
    upload_timeout 10m
    role_arn arn:aws:iam::XXXX:role/role
    sts_endpoint https://sts.xxxx.amazonaws.com
    s3_key_format /opensearch/XXXX/XXXXX/XXXX-$UUID-%M-%S.gz

[OUTPUT]
    Name  opensearch
    Match 11
    Host  opensearch.xxxx
    Port  443
    Index myindex_y%Y_m%m_d%d
    AWS_Auth On
    AWS_Region XXXX
    AWS_Role_ARN  arn:aws:iam::XXXXX:role/XXXX
    tls     On
    Buffer_Size 10M
    Suppress_Type_Name On
    Retry_Limit 10
    compress gzip

Does fluentbit writes data when it reads file ?

graph write increase when activate fluentbit

graph total storage increase when activate fluentbit