S3 output not flushing logs on instance termination #6644
Unanswered
anubhav2212
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I've installed fluent-bit as a Windows service in a Windows Server 2019 EC2 instance. I am using tail as Input from a text file and S3 for Output with upload_timeout as 10 min. If I gracefully shut down the EC2 instance(eg. from AWS console) before the completion of 10 mins, I don't see any logs in S3. As per https://docs.fluentbit.io/manual/pipeline/outputs/s3#reliability, I assumed fluent-bit will send the logs before it shuts down. I've tried both multipart upload and put object. Also tried changing the "Service" config grace period to 20 seconds but no luck. I don't want to keep the upload_timeout very less as it will end up creating lots of files in S3 which will be difficult to analyze later. Do we have any other option which we can use to achieve this?
Config I am using ->
[SERVICE]
flush 5
grace 20
.....
[INPUT]
Name tail
Path {TextFilePath}
Read_from_Head On
[OUTPUT]
Name s3
Match *
bucket {SomeS3Bucket}
region us-east-2
store_dir {SomeLocalDirectory}
upload_timeout 10m
#use_put_object true -> Tried with and without this setting
Beta Was this translation helpful? Give feedback.
All reactions