Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Determine if a design proposal is needed for serializing state snapshots to disk #13764

Open
2 tasks
Tracked by #14395
poulok opened this issue Jun 10, 2024 · 1 comment · May be fixed by #16090
Open
2 tasks
Tracked by #14395

Determine if a design proposal is needed for serializing state snapshots to disk #13764

poulok opened this issue Jun 10, 2024 · 1 comment · May be fixed by #16090
Assignees
Labels
Platform Tickets pertaining to the platform

Comments

@poulok
Copy link
Member

poulok commented Jun 10, 2024

There is some data in the state that is not currently represented in protobuf, such as the virtual node cache.

This should be implemented after the migration of state to a single virtual map to eliminate wasted effort (defining protobuf for parts of the tree that will no longer exist once we migrate to a single virtual map).

It may be possible that virtual node cache never needs to be serialized if it can be guaranteed that the virtual node cache is empty on every snapshot. Most of the work of this ticket is determining if this approach is feasible. If state snapshots do not need to be very fast, this is a good option. If it is still required for snapshots to be very fast, then the virtual node cache will have to be serialized to disk in protobuf format and this design proposal will be more complex. Part of the consideration should be the amount of time it would take to serialize the cache vs. flushing it.

  • Determine the requirements for speed of state snapshots
  • Gather performance data for flushing the virtual node cache vs. serializing it to protobuf

This ticket is complete when the decision of whether or not we will flush the virtual node cache on every snapshot or serialize it.

@poulok poulok added the Platform Tickets pertaining to the platform label Jun 10, 2024
This was referenced Aug 27, 2024
@poulok poulok changed the title Design proposal for merkle tree serialization to protobuf Determine if a design proposal is needed for serializing state snapshots to disk Sep 16, 2024
@artemananiev
Copy link
Member

I checked current metrics in mainnet and testnet, in particular "copy flush time, ms". None of all virtual maps take more than 500ms to flush on any node

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Platform Tickets pertaining to the platform
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants