bug: intense execution of storage erase in waku-sync

[Ivansete-status]

Problem

Intense waku-sync operation that makes the node irresponsive

Impact

Other protocols, such as store cannot operate as usual.

To reproduce

Enable store-sync in status.staging fleet and then it appears an important number of logs with "storage erase failed"

Screenshots/logs

Evidence of the issue in Kibana:

https://kibana.infra.status.im/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:'2024-10-09T00:19:07.466Z',to:'2024-10-18T08:31:50.066Z'))&_a=(columns:!(logsource,message),filters:!(),hideChart:!f,index:ffcc22b0-0116-11ed-9719-cdd3b483481c,interval:auto,query:(language:kuery,query:'fleet:%20%22status.staging%22%20AND%20message:%20%22storage%20erase%20failed%22'),sort:!(!('@timestamp',desc)))

nwaku version/commit hash

v0.33.1

[SionoiS]

The errors are harmless. They come from trying to erase msgs we don't have.

As for the DB load, we could check if Waku Sync is doing more queries than needed (one every 5m max).