Skip to content

Commit

Permalink
[kv store] basic bigtable client (#19890)
Browse files Browse the repository at this point in the history
## Description 
Initial implementation of a KV store on top of BigTable.
notes:
* includes instructions for setting up a local BigTable instance via the
emulator and a basic script for data ingestion(later will be added as an
option to main ingestion binary)
* generated proto files are included in the repo, so users won't have to
compile the definitions themselves or install the `protoc` dependency.
We can add an optional build step in a follow-up PR


---

## Release notes

Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.

For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.

- [ ] Protocol: 
- [ ] Nodes (Validators and Full nodes): 
- [ ] Indexer: 
- [ ] JSON-RPC: 
- [ ] GraphQL: 
- [ ] CLI: 
- [ ] Rust SDK:
- [ ] REST API:
  • Loading branch information
phoenix-o authored Oct 24, 2024
1 parent cf2a573 commit df3cc8b
Show file tree
Hide file tree
Showing 15 changed files with 5,167 additions and 1 deletion.
52 changes: 52 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,7 @@ members = [
"crates/sui-json-rpc-tests",
"crates/sui-json-rpc-types",
"crates/sui-keys",
"crates/sui-kvstore",
"crates/sui-light-client",
"crates/sui-macros",
"crates/sui-metric-checker",
Expand Down Expand Up @@ -346,6 +347,7 @@ futures-core = "0.3.21"
git-version = "0.3.5"
glob = "0.3.1"
governor = "0.6.0"
gcp_auth = "0.12.3"
hashbrown = "0.12"
hdrhistogram = "7.5.1"
hex = "0.4.3"
Expand Down Expand Up @@ -420,6 +422,7 @@ proptest = "1.1.0"
proptest-derive = "0.3.0"
prost = "0.13"
prost-build = "0.13"
prost-types = "0.13.1"
protobuf = { version = "2.28", features = ["with-bytes"] }
quinn-proto = "0.11.7"
quote = "1.0.23"
Expand Down
25 changes: 25 additions & 0 deletions crates/sui-kvstore/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
[package]
name = "sui-kvstore"
authors = ["Mysten Labs <[email protected]>"]
license = "Apache-2.0"
publish = false
edition = "2021"
version.workspace = true

[dependencies]
anyhow.workspace = true
async-trait.workspace = true
base64.workspace = true
bcs.workspace = true
http.workspace = true
gcp_auth.workspace = true
prometheus.workspace = true
prost.workspace = true
prost-types.workspace = true
serde.workspace = true
sui-data-ingestion-core.workspace = true
sui-types.workspace = true
telemetry-subscribers.workspace = true
tokio = { workspace = true, features = ["full"] }
tonic = {version = "0.12.2",features = ["tls", "transport"] }
tracing.workspace = true
16 changes: 16 additions & 0 deletions crates/sui-kvstore/src/bigtable/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
## Setup

### Local development
- install the `cbt` CLI tool
```sh
gcloud components install cbt
```
- start the emulator
```sh
gcloud beta emulators bigtable start
```
- set `BIGTABLE_EMULATOR_HOST` environment variable
```sh
$(gcloud beta emulators bigtable env-init)
```
- Run `./src/bigtable/init.sh` to configure the emulator
Loading

0 comments on commit df3cc8b

Please sign in to comment.