"Library Log File" Grows With No Limit (2.0)

[winzig]

As requested, opening this ticket that was previously closed prematurely to re-report this issue.

The issue: if there is a problem preventing the logs from being able to be sent to CloudWatch, the aws-logger-errors.txt file will be filled and grow with no bounds, eventually filling the disk.

As sattpat replied back in 2019:

We certainly dont want to be logging every single issue. We would instead like to capture exceptions that are unique. Meaning that if you get exceptions as a result of network connection, we only log that once and we skip the repeats.

This seems like a very good solution. Perhaps whatever is logging to aws-logger-errors.txt could keep track of recently written errors, and if it's logging the same error several times in a row, it logs an additional "This issue won't be logged again until the process is restarted", and then stops logging that same error again.

[ashishdhingra]

@winzig Thanks for opening the issue. I'm not sure if logging specific issue only once is feasible. But I will discuss with the team and post any updates here.

[normj]

I think the solution would need to be a bit more sophisticated then that. If we stop logging how does a user know if the problem is still happening or if the process has recovered. So we would need something where we detect we are continually getting problems and if so write that there have been constant failures and they will no longer be written out. And if the failures have stopped we would need to write that the system has corrected itself.

[normj]

Or would it just be easier and sufficient to implement a simple rolling log file approach with a max number of files?

[winzig]

@normj Yeah that would be a great solution I think!

[Alex6015]

Hi,
Any updates on this issue?
we didnt noticed and one of our servers reached 66GB aws-logger-errors.txt file size

[github-actions]

We have noticed this issue has not received attention in over a year. We will close this issue for now. If you think this is in error, please feel free to comment and reopen the issue.

[winzig]

I think this should remain open unless @normj's recommendation of a rolling log file approach has been implemented in the last year?

[udlose]

Any updates on getting this fixed? My ECS Containers are out of space because of this error...