gear-reply-cli create-snapshot uses too much memory

[gshep]

File Location(s)

No response

Proposal

While trying to create a state snapshot for the block 11_118_796 of the testnext, the utility was killed by the OS memory manager. It would be more convenient for the end user to limit the memory usage of gear-replay-cli when creating snapshots.

[ 6808.212686] Out of memory: Killed process 119826 (gear-replay-cli) total-vm:1
5167424kB, anon-rss:11575028kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:2
2888kB oom_score_adj:0

[Lazark0x]

It seems that the options for optimizing memory consumption are rather limited on our part: the create-snapshot argument only configures Builder from crate frame-remote-externalities

Local testing shows that about ~3GB is consumed while downloading key values:

And about ~10GB during DB generation:

The following arguments have been used:
gear-replay-cli create-snapshot -u wss://testnet-archive.vara.network:443 ../test-snapshot -b 0x61e939c1dcf24b7edf58ede944bdf544f65d9c7b2a3757bb2228dfecfd07fec5