Skip to content

Commit

Permalink
Remove server details
Browse files Browse the repository at this point in the history
  • Loading branch information
DOBEN committed Jul 22, 2024
1 parent 43a97ad commit e08c5ac
Show file tree
Hide file tree
Showing 8 changed files with 11 additions and 683 deletions.
2 changes: 1 addition & 1 deletion compliant-reward-distribution/indexer/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@

## 0.1.0

- Initial `indexer` and `server`.
- Initial `indexer`.
69 changes: 1 addition & 68 deletions compliant-reward-distribution/indexer/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 1 addition & 3 deletions compliant-reward-distribution/indexer/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "indexer"
version = "0.2.0"
version = "0.1.0"
edition = "2021"

[dependencies]
Expand All @@ -22,12 +22,10 @@ tower-http = { version = "0.4", features = [
http = "0.2"
tonic = { version = "0.10", features = ["tls-roots", "tls"] }
thiserror = "1.0"
chrono = "0.4"
concordium-rust-sdk = { version = "4.2"}
tokio = { version = "1.35", features = ["rt-multi-thread", "macros", "sync", "signal"] }
tokio-postgres = { version = "0.7", features = [
"with-serde_json-1",
"with-chrono-0_4",
] }
deadpool-postgres = "0.11"
handlebars = "4.5"
60 changes: 5 additions & 55 deletions compliant-reward-distribution/indexer/README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,5 @@
## Compliant-Reward-Distribution Indexer

There are two binaries in this project. An `indexer` that indexes data into a database and a `server` that serves data from the database.

The easiest way to run the `indexer` and `server` is to use [docker-compose](https://docs.docker.com/compose/) as described in the [README.md](../README.md) file.

## Prerequisites

- `PostgreSQL` installed or running it in a docker container: https://www.postgresql.org/download/
Expand All @@ -24,7 +20,7 @@ Alternatively, you can run the Postgres database in a docker container. The comm
docker run -p 5432:5432 -e POSTGRES_PASSWORD=password -e POSTGRES_DB="indexer" --rm postgres
```

## Build the `indexer` and `server`
## Build the `indexer`

To build the tools make sure you have the repository submodules initialized

Expand All @@ -38,19 +34,17 @@ The tool can be built by running
cargo build --release
```

This will produce two binaries (`indexer` and `server`) in the `target/release` directory.
This will produce the binaries (`indexer`) in the `target/release` directory.

# The `indexer` binary

It is a tool for indexing event data from the track and trace contract into a postgres database. The database is configured with the tables from the file `../resources/schema.sql`. The monitored events `ItemStatusChangedEvent` and `ItemCreatedEvent` are indexed in their respective tables. A third table `settings` exists to store global configurations (e.g.: the contract address, latest block processed, and the genesis block hash).
It is a tool for indexing newly created accounts on Concordium into a postgres database. The database is configured with the tables from the file `../resources/schema.sql`. A table `settings` exists to store global configurations.

The global configurations are set when the indexer is started for the first time. Re-starting the indexer will check if its current settings are compatible will the stored indexer settings to prevent corrupting the database. In addition, the settings can be queried by the front end to check compatibility.

When the indexer is started for the first time, it will look up when the smart contract instance was created and use that block as the starting block. When the indexer is re-started with the same database settings, it resumes indexing from the `latest_processed_block_height+1` as stored in the database.
When the indexer is started for the first time, it will look up the current block height and start indexing from that block. When the indexer is re-started with the same database settings, it resumes indexing from the `latest_processed_block_height+1` as stored in the database.

All monitored events in a block are atomically added in one database transaction to postgres. This ensures a simple recovery process since we always process the complete block or roll back the database to the beginning of the block. In addition, the indexer has a re-try logic and will try to re-connect to the database pool and re-submit any failed database transaction.

Each event can be uniquely identified by the `transaction_hash` and `event_index`. The `event_index` is the index from the array of logged events in a transaction.
All newly created accounts in a block are atomically added in one database transaction to postgres. This ensures a simple recovery process since we always process the complete block or roll back the database to the beginning of the block. In addition, the indexer has a re-try logic and will try to re-connect to the database pool and re-submit any failed database transaction.

## Run the `indexer`

Expand All @@ -67,47 +61,3 @@ There are a few options to configure the indexer:
- `--db-connection` should specify your postgreSQL database connection. If not specified, the default value `host=localhost dbname=indexer user=postgres password=password port=5432` is used.

- `--log-level` specifies the maximum log level. Possible values are: `trace`, `debug`, `info`, `warn`, and `error`. If not specified, the default value `info` is used.

## The `server` binary

You have to build the front end in the folder `../frontend` before running this command.

## Run the `server`

```console
cargo run --bin server
```

## Configure the `server`

There are a few options to configure the server:

- `--listen-address` is the listen address where the server will be listen on. If not specified, the default value `0.0.0.0:8080` is used.

- `--frontend` is the path to the directory where the frontend assets are located. If not specified, the default value `../frontend/dist` is used.

- `--db-connection` should specify your postgreSQL database connection. If not specified, the default value `host=localhost dbname=indexer user=postgres password=password port=5432` is used.

- `--log-level` specifies the maximum log level. Possible values are: `trace`, `debug`, `info`, `warn`, and `error`. If not specified, the default value `info` is used.

The following option are also available, which are forwarded to the frontend:

- `--node` specifies the gRPC interface of a Concordium node. (Defaults to `https://grpc.testnet.concordium.com:20000`)

- `--network` specifies the network to use, i.e., `mainnet` or `testnet`. Defaults to `testnet`.

- `--contract-address` specifies the contract address of the track and trace contract (format is `<1234,0>`).

- `--sponsored-transaction-backend` specifies the endpoint to the sponsored transaction backend. (Defaults to `http://localhost:8000`).

An example of running the service with basic settings and testnet node would be:

``` console
cargo run --bin server -- --contract-address <YOUR_CONTRACT_ADDRESS>
```

An example to run the service with some filled in example settings would be:

``` console
cargo run --bin server -- --contract-address "<8901,0>"
```
52 changes: 1 addition & 51 deletions compliant-reward-distribution/indexer/resources/schema.sql
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ CREATE TABLE IF NOT EXISTS settings (
-- Re-starting the indexer will check if its settings are compatible with
-- the stored indexer setting to prevent corrupting the database.
genesis_block_hash BYTEA NOT NULL,
-- Start block that was indexed.
-- Start block height that was indexed.
start_block_height INT8,
-- The last block height that was processed.
latest_processed_block_height INT8
Expand Down Expand Up @@ -37,53 +37,3 @@ CREATE TABLE IF NOT EXISTS accounts (
-- claiming with different accounts for the same identity.
uniqueness_hash BYTEA
);

CREATE TABLE IF NOT EXISTS item_status_changed_events (
-- Primary key.
id INT8 PRIMARY KEY,
-- The timestamp of the block the event was included in.
block_time TIMESTAMP WITH TIME ZONE NOT NULL,
-- The transaction hash that the event was included in.
transaction_hash BYTEA NOT NULL,
-- The index from the array of logged events in a transaction.
event_index INT8 NOT NULL,
-- The item's id as logged in the event.
item_id INT8 NOT NULL
);

-- Table containing item_status_changed_events successfully submitted to the database from the contract monitored.
CREATE TABLE IF NOT EXISTS item_status_changed_events (
-- Primary key.
id INT8 PRIMARY KEY,
-- The timestamp of the block the event was included in.
block_time TIMESTAMP WITH TIME ZONE NOT NULL,
-- The transaction hash that the event was included in.
transaction_hash BYTEA NOT NULL,
-- The index from the array of logged events in a transaction.
event_index INT8 NOT NULL,
-- The item's id as logged in the event.
item_id INT8 NOT NULL
);

-- Table containing item_created_events successfully submitted to the database from the contract monitored.
CREATE TABLE IF NOT EXISTS item_created_events (
-- Primary key.
id INT8 PRIMARY KEY,
-- The timestamp of the block the event was included in.
block_time TIMESTAMP WITH TIME ZONE NOT NULL,
-- The transaction hash that the event was included in.
transaction_hash BYTEA NOT NULL,
-- The index from the array of logged events in a transaction.
event_index INT8 NOT NULL,
-- The item's id as logged in the event.
item_id INT8 NOT NULL,
-- The item's metadata_url as logged in the event.
metadata_url BYTEA NOT NULL
);

-- -- Improve performance on queries for events with given item_id.
-- CREATE INDEX IF NOT EXISTS item_changed_index ON item_status_changed_events (item_id);
-- -- Improve performance on queries for events with given current status.
-- CREATE INDEX IF NOT EXISTS current_status_index ON item_status_changed_events (new_status);
-- -- Improve performance on queries for events with given item_id.
-- CREATE INDEX IF NOT EXISTS item_created_index ON item_created_events (item_id);
2 changes: 1 addition & 1 deletion compliant-reward-distribution/indexer/src/bin/indexer.rs
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@ impl indexer::ProcessEvent for StoreEvents {
)?;

tracing::debug!(
"Preparing database transaction for account {} from transaction hash {} \
"Preparing database transaction for account {:} from transaction hash {} \
in block {}.",
account_creation_details.address,
tx.hash,
Expand Down
Loading

0 comments on commit e08c5ac

Please sign in to comment.