Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(codecv1): support zstd compression #1288

Closed
wants to merge 2 commits into from

Conversation

colinlyguo
Copy link
Member

@colinlyguo colinlyguo commented Apr 21, 2024

Purpose or design rationale of this PR

Use https://github.com/DataDog/zstd as a base repo, and:

  1. add CompressScrollBatchBytes in this commit.
  2. to align with rust side, upgrade c zstd version to v1.5.6 in this commit, reusing scripts in: Update vendored zstd to 1.5.5 DataDog/zstd#125.

PR title

Your PR title must follow conventional commits (as we are doing squash merge for each PR), so it must start with one of the following types:

  • feat: A new feature

Deployment tag versioning

Has tag in common/version.go been updated or have you added bump-version label to this PR?

  • Yes

Breaking change label

Does this PR have the breaking-change label?

  • No, this PR is not a breaking change

@codecov-commenter
Copy link

codecov-commenter commented Apr 21, 2024

Codecov Report

Attention: Patch coverage is 63.15789% with 7 lines in your changes are missing coverage. Please review.

Project coverage is 56.07%. Comparing base (69c0f7e) to head (4240be9).
Report is 3 commits behind head on develop.

Files Patch % Lines
common/types/encoding/codecv1/codecv1.go 63.15% 5 Missing and 2 partials ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #1288      +/-   ##
===========================================
- Coverage    56.11%   56.07%   -0.05%     
===========================================
  Files          156      156              
  Lines        12131    12156      +25     
===========================================
+ Hits          6807     6816       +9     
- Misses        4798     4809      +11     
- Partials       526      531       +5     
Flag Coverage Δ
bridge-history-api 72.50% <ø> (ø)
common 61.90% <63.15%> (-0.22%) ⬇️
coordinator 19.93% <ø> (ø)
database 42.85% <ø> (ø)
prover 26.55% <ø> (ø)
rollup 62.27% <ø> (-0.09%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@colinlyguo colinlyguo force-pushed the feat-codecv1-add-zstd-compression branch 4 times, most recently from c170767 to 6a72f17 Compare April 22, 2024 16:31
@colinlyguo colinlyguo force-pushed the feat-codecv1-add-zstd-compression branch 3 times, most recently from 277b0be to ffe2023 Compare April 22, 2024 16:59
@colinlyguo colinlyguo force-pushed the feat-codecv1-add-zstd-compression branch 2 times, most recently from 1b97dac to 0a46938 Compare April 23, 2024 16:43
@colinlyguo colinlyguo force-pushed the feat-codecv1-add-zstd-compression branch from 21bfc67 to 4240be9 Compare April 23, 2024 17:26
@@ -10,6 +10,7 @@ import (
"math/big"
"strings"

"github.com/colinlyguo/zstd"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should move this repo to scroll later?

@0xmountaintop
Copy link
Member

we should review colinlyguo/zstd@d3f94bd and colinlyguo/zstd@fcb551e first

@colinlyguo
Copy link
Member Author

we should review colinlyguo/zstd@d3f94bd and colinlyguo/zstd@fcb551e first

we can initialize a new repo under scroll-tech (forked from https://github.com/DataDog/zstd), and I can move these commits to it, for review.

@colinlyguo
Copy link
Member Author

discussed with @georgehao, we can package a zstd rust .so lib with functions like compress_scroll_batch_bytes, better supporting future zstd upgrades.

@colinlyguo colinlyguo changed the title feat(codecv1): add zstd compression feat(codecv1): support zstd compression Apr 26, 2024
@@ -204,7 +205,7 @@ func (c *DAChunk) Hash() (common.Hash, error) {
}

// NewDABatch creates a DABatch from the provided encoding.Batch.
func NewDABatch(batch *encoding.Batch) (*DABatch, error) {
func NewDABatch(batch *encoding.Batch, withCompression bool) (*DABatch, error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to make sure I understand correctly: when withCompression = true, the only things that change are: blob, BlobVersionedHash, right? So z, DataHash etc. won't change.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

z would be changed because z is influenced by blobVersionedHash.

@@ -10,6 +10,7 @@ import (
"math/big"
"strings"

"github.com/colinlyguo/zstd"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you plan to change this before merging this PR?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes.

@colinlyguo
Copy link
Member Author

@colinlyguo colinlyguo closed this Apr 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants