Restarting cache nodes can exhaust disk space

[bryanlb]

Describe the bug

In the event a cache node is restarted but not recreated/destroyed, the previously cached data can still exist on disk. This can cause out-of-disk errors as these old chunks have no reference but still take up disk space.

To Reproduce

Restart a running cache node, and after reboot observe the KALDB_CACHE_DATA_DIR has chunks that were created prior to the reboot time.

Expected behavior

The simplest solution here would be to delete the existing chunks on boot. Note that we should consider what happens when a user attempts to map the data directory to a shared folder. We may consider storing all chunks in a sub-directory of the configured data directory, and then deleting/re-creating that specific folder.

[bryanlb]

This is still an issue, however we're currently working around this by clearing the directory in our entrypoint.sh script with something like the following, where we use /astra_data as the data directory:

# Temporary fix for reboots leaving cache assets behind
echo "Clearing /astra_data directory"
rm -rfv /astra_data/*