[SMST] feat: Use compact SMST proofs

[red-0ne]

Summary

This PR changes the proofs being sent in MsgSubmitPRoof and stored by the SubmitProof keeper from SparseMerkleClosestProof to SparseCompactMerkleClosestProof.

Issue

Proofs one of the most block-space consuming primitives. The SMST support compact proofs that could help reduce their sizes.

• [Load Testing] Permissionless Demand Implementation Umbrella Ticket  #758

Type of change

Select one or more from the following:

• New feature, functionality or library
• Consensus breaking; add the consensus-breaking label if so. See [Infra] Automatically add the consensus-breaking label #791 for details
• Bug fix
• Code health or cleanup
• Documentation
• Other (specify)

Testing

• Unit Tests: make go_develop_and_test
• LocalNet E2E Tests: make test_e2e
• DevNet E2E Tests: Add the devnet-test-e2e label to the PR.

Sanity Checklist

• I have tested my changes using the available tooling
• I have commented my code
• I have performed a self-review of my own code; both comments & source code

[Olshansk]

Amazed that everyone just worked!

Any chance you can add a log or a unit test to show the difference in size?

I'm REALLy curious and can be done in a followup.

[red-0ne]

Any chance you can add a log or a unit test to show the difference in size?

@Olshansk, I'm really grateful that you called this out.

I added this test /pkg/relayer/session/sessiontree_test.go

Which shows that there is barely any improvement in the proof size 😢

I added a (gzip) compressed proofs to the test to have something to compare with.

Running for different leaf sizes, 1000 run each and accumulating size, this a sample result

numLeaf=10: compactionRatio: 0.990917, compressionRatio: 0.804937
numLeaf=100: compactionRatio: 0.989091, compressionRatio: 0.803551
numLeaf=1000: compactionRatio: 0.992390, compressionRatio: 0.805643
numLeaf=10000: compactionRatio: 0.989311, compressionRatio: 0.803930
numLeaf=100000: compactionRatio: 0.992038, compressionRatio: 0.805471
numLeaf=1000000: compactionRatio: 0.990228, compressionRatio: 0.804995

[Olshansk]

@red-0ne Thanks for putting in the extra test. This is fine and what I expected.

Want to use this as a learning lesson. Nothing we don't already know but something I want to reiterate:

1. Compact proof sounds sexy, eh? This is a classic example of "sounds sexy to an engineer" but not actually useful.
   • Probabilistic proof is what will actually ensure scalability
2. Understanding what's happening under the hood
   • We inherited "Compact Proofs" from the library we forked
   • I didn't implement it, neither did Harry, neither did you
   • None of us looked in the source code
   • I'm 100% sure we can make it more efficient, but again, it "sounds sexy", right?
3. The importance of visibility / observability
   • This doesn't always need to be a dashboard
   • It doesn't always need to be telemetry
   • It could be a log or a unit test
   • This is a great example of using t.Log
   • This raises the importance of a dashboard for probabilistic proofs

[github-actions]

The CI will now also run the e2e tests on devnet, which increases the time it takes to complete all CI checks.

You may need to run make trigger_ci to submit an empty commit that'll trigger the tests.

GCP workloads (requires changing the namespace to 823)
Grafana network dashboard for devnet-issue-823

[h5law]

@red-0ne Thanks for putting in the extra test. This is fine and what I expected.

Want to use this as a learning lesson. Nothing we don't already know but something I want to reiterate:

1. Compact proof sounds sexy, eh? This is a classic example of "sounds sexy to an engineer" but not actually useful.

   • Probabilistic proof is what will actually ensure scalability

2. Understanding what's happening under the hood

   • We inherited "Compact Proofs" from the library we forked
   • I didn't implement it, neither did Harry, neither did you
   • None of us looked in the source code
   • I'm 100% sure we can make it more efficient, but again, it "sounds sexy", right?

3. The importance of visibility / observability

   • This doesn't always need to be a dashboard
   • It doesn't always need to be telemetry
   • It could be a log or a unit test
   • This is a great example of using t.Log
   • This raises the importance of a dashboard for probabilistic proofs

Little late to this but just going through my notifications. I actually did implement the CompactClosestProof type to fit the pattern with the other proof types. Had some brief ideas about how they could be compacted properly (was in a rush when implementing this):

1. Clip the Path field to Depth bits
2. Replace the FlippedBits byte matrix with a bit mask where 1s represent bits flipped, the [][]byte can be replaced with uint{Depth} as each bit represents an index (up to 4 uint64s for a full 256-bit worst case encoding)
3. Instead of encoding the Depth as the []byte form of the original int use a single byte (this assumes a 256 bit hasher or figure out a 9-bit type to generalise and include 512-bit hashers)

Realisticallly this method is super optimisable, I didn't really try to optimise it as it was always don't pre-optimise. I'm sure you guys will think of even more, actually doing compression on the encoded bits would be a interesting idea once they are encoded better.