[Badger] Add universal database operations #6465

zhangchiqing · 2024-09-14T00:53:21Z

This PR implemented the low level database operations that can be shared by both badger and pebble implementation.

Since badger operations will be refactored to use batch updates instead of transactions, this makes the badger db operations and pebble db operations very similar and I managed to unify them.

A separate PR that refactors the approvals will demonstrate how the unified database operations can be used there. (see #6466)

This allows us to remove both the badger-specific and pebble-specific database modules, and only keep one universal version, easier to maintain.

codecov-commenter · 2024-09-14T00:56:48Z

Codecov Report

Attention: Patch coverage is 28.60262% with 327 lines in your changes missing coverage. Please review.

Project coverage is 41.16%. Comparing base (681dd9a) to head (17972e0).

Files with missing lines	Patch %	Lines
storage/operation/badgerimpl/writer.go	0.00%	48 Missing ⚠️
storage/operation/pebbleimpl/writer.go	0.00%	42 Missing ⚠️
storage/operation/reads.go	67.74%	27 Missing and 13 partials ⚠️
storage/operation/badgerimpl/iterator.go	0.00%	33 Missing ⚠️
storage/operation/pebbleimpl/iterator.go	0.00%	33 Missing ⚠️
storage/operation/dbtest/helper.go	0.00%	31 Missing ⚠️
utils/unittest/unittest.go	0.00%	27 Missing ⚠️
storage/operation/badgerimpl/reader.go	0.00%	20 Missing ⚠️
storage/operation/codec.go	27.27%	16 Missing ⚠️
storage/operation/pebbleimpl/reader.go	0.00%	14 Missing ⚠️
... and 3 more

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6465      +/-   ##
==========================================
- Coverage   41.19%   41.16%   -0.04%     
==========================================
  Files        2052     2064      +12     
  Lines      182191   182649     +458     
==========================================
+ Hits        75062    75183     +121     
- Misses     100839   101156     +317     
- Partials     6290     6310      +20

Flag	Coverage Δ
unittests	`41.16% <28.60%> (-0.04%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

zhangchiqing · 2024-09-14T00:55:34Z

storage/operation/reads_test.go

@@ -0,0 +1,189 @@
+package operation_test


Test cases are taken from similar ones in storage/badger/operation/common_test.go

zhangchiqing · 2024-09-14T00:55:45Z

storage/operation/writes_test.go

@@ -0,0 +1,278 @@
+package operation_test


Test cases are taken from similar ones in storage/badger/operation/common_test.go

zhangchiqing · 2024-09-14T00:59:24Z

storage/operation/reads.go

+			keyCopy := make([]byte, len(key))
+			copy(keyCopy, key)


Copying the key is for safety, otherwise caller might make mistake storing the key in a slice and causing problem because this iteration here might overwrite the underlying slice. Making a copy of the key could prevent caller from making such mistake.

zhangchiqing · 2024-09-14T01:01:42Z

storage/operation/badgerimpl/iterator.go

+	tx := db.NewTransaction(false)
+	iter := tx.NewIterator(options)
+
+	lowerBound, upperBound := storage.StartEndPrefixToLowerUpperBound(startPrefix, endPrefix)


Note, we are applying the same approach as pebble to compute the exclusive upperBound here. This allows us to get rid of the global max value stored in the database (see: storage/badger/operation/max.go)

zhangchiqing · 2024-09-14T01:02:45Z

storage/operation/reads_test.go

+		keys := [][]byte{
+			// before start -> not included in range
+			{0x09, 0xff},
+			// within the start prefix -> included in range
+			{0x10, 0x00},
+			{0x10, 0xff},
+			// between start and end -> included in range
+			{0x15, 0x00},
+			{0x1A, 0xff},
+			// within the end prefix -> included in range
+			{0x20, 0x00},
+			{0x20, 0xff},
+			// after end -> not included in range
+			{0x21, 0x00},
+		}


The same test cases to test the boundary cases

zhangchiqing · 2024-09-14T01:03:04Z

cmd/bootstrap/utils/md5.go

 import (
+	// #nosec


fix linter error

zhangchiqing · 2024-09-16T23:07:40Z

storage/operation/pebbleimpl/writer.go

+}
+
+func (b *ReaderBatchWriter) Commit() error {
+	err := b.batch.Commit(pebble.Sync)


With the database operations being universal, it's easy to write benchmark to compare the performance between badger and pebble implementation.

The read performance for badger and pebble are very similar.

BenchmarkRetrieve/BadgerStorage-10 1217998 948.1 ns/op BenchmarkRetrieve/PebbleStorage-10 1699320 725.4 ns/op

However, for writes, the benchmark shows pebble is 100x slower than badger.

I initially did:

cd storage/operation go test -bench=BenchmarkUpsert goos: darwin goarch: arm64 pkg: github.com/onflow/flow-go/storage/operation BenchmarkUpsert/BadgerStorage-10 31804 35173 ns/op BenchmarkUpsert/PebbleStorage-10 270 4267359 ns/op PASS ok github.com/onflow/flow-go/storage/operation 4.886s

I tracked down to this Commit call by adding the following code.

n1 := time.Now() err := b.batch.Commit(pebble.Sync) n2 := time.Now() fmt.Println("pebbleimpl.Writer.go: Commit() time:", n2.Sub(n1).Nanoseconds())

Then I did the same to badger to record the duration for the Commit call, and run the TestReadWrite test case which writes a single entity data to the database, the result shows pebble is 25x slower than badger for writes (read performance is similar). I'm not sure why.

cd storage/operation gt -run=TestReadWrite badgerimpl.Writer.go: Commit() time: 147250 badgerimpl.Writer.go: Commit() time: 71458 badgerimpl.Writer.go: Commit() time: 72708 pebbleimpl.Writer.go: Commit() time: 3684875 pebbleimpl.Writer.go: Commit() time: 3006416 pebbleimpl.Writer.go: Commit() time: 3001083 badgerimpl.Writer.go: Commit() time: 6500 pebbleimpl.Writer.go: Commit() time: 584 PASS ok github.com/onflow/flow-go/storage/operation 1.136s

Thanks for making the benchmarks and finding this regression @zhangchiqing 🙇

100x slower writes is a big difference! 😬

I think it would be worthwhile to spend some time better understanding this finding and investigating how it would impact Flow (for example, the consensus hot-path). My gut feeling is that this benchmark in isolation is overstating the actual impact we would see. If it isn't, and switching to Pebble is going to make all our writes 2 orders of magnitude slower, I'd rather know that now.

Are there Pebble configs we can change to bring write speed down?

Does this result hold if aspects of the environment are changed (eg. maybe it's an artifact of the test database being very small? Just guessing)

Does this result hold if aspects of the load are changed (eg. committing a batch containing many writes)

I spent a short time during the review trying a few different iterations on BenchmarkUpsert:

Setting WithKeepL0InMemory to false (previously was true).

Setting WithValueThreshold to 1 byte (previously was 1kb).

Badger is different from Pebble in that it will sometimes store values directly in the LSM tree, and other times in a value log file, based on the value's size. All the values we were storing were below that 1kb threshold.

Using larger 10kb values

Unfortunately, all of these had similar results where Pebble was still ~100x slower...

Yeah, this might take some time to investigate. I will do it in a separate issue. But it should not block us from getting in with this PR, since it also allows us to move from badger transaction to badger batch updates.

Regarding the regression, Peter suspect it might be to do with the key-compression and decompression, since he did a profiling in the past and saw a lot of CPU cycles are spent on that. So might worth to try disabling compression in pebble and see if it makes a difference.

storage/batch.go

utils/unittest/unittest.go

storage/operations.go

storage/operation/reads_test.go

jordanschalm · 2024-11-01T15:44:32Z

storage/operation/reads_test.go

+
+func TestTraverse(t *testing.T) {
+	dbtest.RunWithStorages(t, func(t *testing.T, r storage.Reader, withWriter dbtest.WithWriter) {
+		keys := [][]byte{


This would probably be easier to work with as a map from key -> value. If we define the key type as [2]byte, and then do key[:] when inserting it, that would work.

Feel free to skip this suggestion.

storage/operation/writes_test.go

jordanschalm

This looks great. I like the way you structured testing so we can easily test both database backends together 💯

Besides the suggestions above, could you:

Make sure exported types have at least a basic godoc (copying the interface documentation to structs that implement it works). I added suggestions about this in a few places, but not all.
Make sure all public functions that return errors document any expected error types, or that "no errors are expected during normal operation".

jordanschalm

🎸

jordanschalm · 2024-11-04T16:54:35Z

storage/operations.go

+}
+
+// Writer is an interface for batch writing to a storage backend.
+// It cannot be used concurrently for writing.


Suggested change

// It cannot be used concurrently for writing.

// One Writer instance cannot be used concurrently by multiple goroutines.

// However, many goroutines can write concurrently, each using their own Writer instance.

zhangchiqing commented Sep 14, 2024

View reviewed changes

zhangchiqing mentioned this pull request Sep 14, 2024

[Badger] Refactor approvals to use badger batch updates #6466

Draft

zhangchiqing commented Sep 16, 2024

View reviewed changes

zhangchiqing marked this pull request as ready for review September 16, 2024 23:15

zhangchiqing requested review from jordanschalm, AlexHentschel, peterargue and durkmurder September 16, 2024 23:16

j1010001 mentioned this pull request Oct 3, 2024

Replace Badger with Pebble DB - Execution data #6527

Open

zhangchiqing force-pushed the leo/db-ops branch from 87dc8e1 to b82106a Compare October 8, 2024 20:40

zhangchiqing force-pushed the leo/db-ops branch from 5c99655 to 257bde0 Compare October 28, 2024 23:44

This was referenced Oct 28, 2024

[Storage] Pruning - add a test case to verify compaction can reclaim disk space #6603

Open

[Review Version] Refactor Protocol State with Pebble-based Storage #6197

Closed

zhangchiqing force-pushed the leo/db-ops branch from 257bde0 to bbfb3a1 Compare October 30, 2024 16:56

jordanschalm reviewed Nov 1, 2024

View reviewed changes

zhangchiqing force-pushed the leo/db-ops branch 4 times, most recently from 284750a to 93fcaa1 Compare November 1, 2024 21:45

zhangchiqing added 4 commits November 1, 2024 17:06

add universal database operations

8d98f0f

address review comments

90a2426

consolidate PrefixUpperbound

326772e

update comments

17972e0

zhangchiqing force-pushed the leo/db-ops branch from 77b44d1 to 17972e0 Compare November 2, 2024 00:06

zhangchiqing requested a review from jordanschalm November 2, 2024 00:06

jordanschalm approved these changes Nov 4, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Badger] Add universal database operations #6465

[Badger] Add universal database operations #6465

zhangchiqing commented Sep 14, 2024 •

edited

Loading

codecov-commenter commented Sep 14, 2024 •

edited

Loading

zhangchiqing Sep 14, 2024

zhangchiqing Sep 14, 2024

zhangchiqing Sep 14, 2024

zhangchiqing Sep 14, 2024

zhangchiqing Sep 14, 2024

zhangchiqing Sep 14, 2024

zhangchiqing Sep 16, 2024 •

edited

Loading

jordanschalm Nov 1, 2024

zhangchiqing Nov 1, 2024

jordanschalm Nov 1, 2024

jordanschalm left a comment

jordanschalm left a comment

jordanschalm Nov 4, 2024

	// It cannot be used concurrently for writing.
	// One Writer instance cannot be used concurrently by multiple goroutines.
	// However, many goroutines can write concurrently, each using their own Writer instance.

[Badger] Add universal database operations #6465

Are you sure you want to change the base?

[Badger] Add universal database operations #6465

Conversation

zhangchiqing commented Sep 14, 2024 • edited Loading

codecov-commenter commented Sep 14, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhangchiqing Sep 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jordanschalm left a comment

Choose a reason for hiding this comment

jordanschalm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhangchiqing commented Sep 14, 2024 •

edited

Loading

codecov-commenter commented Sep 14, 2024 •

edited

Loading

zhangchiqing Sep 16, 2024 •

edited

Loading