feat: make a txsim binary build from Makefile and dockerfile #2052

Bidon15 · 2023-07-10T08:57:13Z

Closes https://github.com/celestiaorg/devops/issues/249

docker/txsim.sh

codecov-commenter · 2023-07-10T09:07:29Z

Codecov Report

Merging #2052 (b1e0a68) into main (83aff91) will increase coverage by 0.02%.
The diff coverage is 25.00%.

❗ Current head b1e0a68 differs from pull request most recent head 6f5204b. Consider uploading reports for the commit 6f5204b to get more accurate results

@@            Coverage Diff             @@
##             main    #2052      +/-   ##
==========================================
+ Coverage   25.02%   25.05%   +0.02%     
==========================================
  Files         121      121              
  Lines       13806    13811       +5     
==========================================
+ Hits         3455     3460       +5     
  Misses       9995     9995              
  Partials      356      356

Impacted Files	Coverage Δ
x/blob/types/payforblob.go	`73.55% <25.00%> (+0.65%)`	⬆️

docker/Dockerfile_txsim

Makefile

.github/workflows/docker-build-publish.yml

docker/README.md

Co-authored-by: Jose Ramon Mañes <[email protected]>

tty47

LGTM! 🚀

Makefile

docker/Dockerfile_txsim

docker/README.md

rootulp · 2023-07-17T18:47:15Z

docker/README.md

@@ -0,0 +1,122 @@
+# Celestia `txsim` Docker Image Usage Guide
+
+The `txsim` binary is a tool that can be used to simulate transactions on the Celestia network. It can be used to test the performance of the Celestia network.


[no change needed][follow up issue] there are other Dockerfiles in this directory. They weren't touched in this PR but maintainers should consider adding to this README what the other files do (i.e. Dockerfile_ephemeral)

Yes. Ideally, we need to make a dedicated issue to document other 2 DockerFiles we have in the repo.
I started this ReadMe, so we can spec out not only the protocol, but what options do we have to run the software apart from vanilla make install && celestia-appd xyz in the host machine

rootulp · 2023-07-17T18:54:44Z

docker/README.md

+## Flag Breakdown
+
+Here's a breakdown of what each flag means:
+
+- `-k`: Whether a new key should be created (1 for yes, 0 for no)
+- `-p`: The path to the keyring for the prefunded account
+- `-r`: The RPC endpoints for the `txsim` binary
+- `-g`: The gRPC endpoints for the `txsim` binary
+- `-t`: The poll time for the `txsim` binary
+- `-b`: The number of blob sequences to run
+- `-a`: The range of blobs to send per PFB in a sequence
+- `-s`: The range of blob sizes to send
+- `-m`: The mnemonic for the keyring
+- `-d`: The seed for the random number generator
+- `-e`: The number of send sequences to run
+- `-i`: The amount to send from one account to another
+- `-v`: The number of send iterations to run per sequence
+- `-u`: The number of stake sequences to run
+- `-w`: The amount of initial stake per sequence


ahh this legend really helps. It may be even clearer to use longer format flags. I.e. --seed, --key, etc. but not blocking

Unfortunately, getopts doesn't support long-option flagging and writing a custom for-loop with edge cases of optional arguments is not what I really want to do.

As a v2, I'd like to remove completely this script and make it as server with REST APIs that we can trigger via a dashboard.

docker/txsim.sh

rootulp · 2023-07-17T18:58:11Z

docker/README.md

+You can run the `txsim` Docker image using the `docker run` command. Here's an example:
+
+```bash
+docker run -it -v ${HOME}/.celestia-app:/home/celestia ghcr.io/celestiaorg/txsim -k 0 -r http://consensus-validator-robusta-rc6.celestia-robusta.com:26657,http://consensus-full-robusta-rc6.celestia-robusta.com:26657 -g consensus-validator-robusta-rc6.celestia-robusta.com:9090 -t 10s -b 10 -d 100 -e 10


[optional] I'm a Docker noob so I don't know what these shorthand flags do. The long format flags would help a lot for readability

Motivation: https://changelog.com/posts/use-long-flags-when-scripting

For example --volume instead of -v

If I get rid of getopts, then yes, we will have long-format flags.
But I want to pursue a simple server approach that can be triggered on/off rather than a cli command 😅

[still not blocking][optional] I don't see why getopts is related to this proposal. The docker run --help docs claim that --volume is the long form for -v

-v, --volume list Bind mount a volume

and I guess -it can be replaced with

-i, --interactive Keep STDIN open even if not attached -t, --tty Allocate a pseudo-TTY

but I think maybe the remaining flags are unique to the script running inside the docker container so they can't be replaced with longer format flags.

Suggested change

docker run -it -v ${HOME}/.celestia-app:/home/celestia ghcr.io/celestiaorg/txsim -k 0 -r http://consensus-validator-robusta-rc6.celestia-robusta.com:26657,http://consensus-full-robusta-rc6.celestia-robusta.com:26657 -g consensus-validator-robusta-rc6.celestia-robusta.com:9090 -t 10s -b 10 -d 100 -e 10

docker run --interactive --tty --volume ${HOME}/.celestia-app:/home/celestia ghcr.io/celestiaorg/txsim -k 0 -r http://consensus-validator-robusta-rc6.celestia-robusta.com:26657,http://consensus-full-robusta-rc6.celestia-robusta.com:26657 -g consensus-validator-robusta-rc6.celestia-robusta.com:9090 -t 10s -b 10 -d 100 -e 10

Co-authored-by: Rootul P <[email protected]>

rootulp

Thanks for the contribution @Bidon15 ! All my remaining feedback in this PR is optional so thanks for creating GH issues! FWIW I haven't tested this manually.

rootulp · 2023-07-18T14:32:37Z

Makefile

+
+## docker-txsim-build: Build the Docker image tx simulator.
+txsim-build-docker:
+	docker build -t ghcr.io/celestiaorg/txsim -f docker/Dockerfile_txsim  .


[optional]

Suggested change

docker build -t ghcr.io/celestiaorg/txsim -f docker/Dockerfile_txsim .

docker build --tag ghcr.io/celestiaorg/txsim --file docker/Dockerfile_txsim .

evan-forbes

LGTM

MSevey · 2023-07-18T18:07:43Z

.github/workflows/docker-build-publish.yml

+
+  docker-txsim-build:
+    permissions:
+      contents: write
+      packages: write
+    uses: celestiaorg/.github/.github/workflows/[email protected]
+    with:
+      dockerfile: docker/Dockerfile_txsim


Any reason we would just combine and put as a matrix?

docker-security-build: strategy: matrix: dockerfile: [Dockerfile, docker/Dockerfile_txsim] permissions: contents: write packages: write uses: celestiaorg/.github/.github/workflows/[email protected] with: dockerfile: ${{ matrix.dockerfile }}

REF: https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs#defining-the-maximum-number-of-concurrent-jobs

Github will try and run them in parallel, so unless there is a bug this is trying to work around, making them separate doesn't speed things up.

also can be a follow up, doesn't need to block in PR. 👍

evan-forbes

lgtm utack

evan-forbes · 2023-07-24T10:28:27Z

need one of us to merge @Bidon15?

github-actions bot added the external label Jul 10, 2023

Bidon15 mentioned this pull request Jul 10, 2023

txsim: sequence failures cause the simulator to either shut down or continue working #2053

Closed

4 tasks

Bidon15 requested review from smuu and tty47 July 10, 2023 09:01

Bidon15 commented Jul 10, 2023

View reviewed changes

docker/txsim.sh Outdated Show resolved Hide resolved

rootulp assigned Bidon15 Jul 10, 2023

tty47 reviewed Jul 14, 2023

View reviewed changes

docker/Dockerfile_txsim Outdated Show resolved Hide resolved

tty47 reviewed Jul 14, 2023

View reviewed changes

docker/Dockerfile_txsim Outdated Show resolved Hide resolved

tty47 reviewed Jul 14, 2023

View reviewed changes

Makefile Show resolved Hide resolved

Bidon15 force-pushed the txsim-cmd branch from a5a1d70 to 172f5ff Compare July 17, 2023 15:53

Bidon15 requested a review from tty47 July 17, 2023 15:53

Bidon15 marked this pull request as ready for review July 17, 2023 15:53

Bidon15 requested review from evan-forbes, rootulp and cmwaters as code owners July 17, 2023 15:53

Bidon15 added 2 commits July 17, 2023 17:53

feat: make a txsim binary build from Makefile and dockerfile

8618ba0

feat: txsim with flags + README

327f5d5

Bidon15 force-pushed the txsim-cmd branch from 172f5ff to 327f5d5 Compare July 17, 2023 15:53

MSevey requested a review from a team July 17, 2023 15:54

chore: respect new line in .yml action

f1d9c1c

Bidon15 requested a review from MSevey July 17, 2023 15:56

tty47 reviewed Jul 17, 2023

View reviewed changes

.github/workflows/docker-build-publish.yml Outdated Show resolved Hide resolved

tty47 reviewed Jul 17, 2023

View reviewed changes

docker/README.md Outdated Show resolved Hide resolved

tty47 reviewed Jul 17, 2023

View reviewed changes

docker/README.md Outdated Show resolved Hide resolved

Apply suggestions from code review

20c2332

Co-authored-by: Jose Ramon Mañes <[email protected]>

tty47 approved these changes Jul 17, 2023

View reviewed changes

rootulp reviewed Jul 17, 2023

View reviewed changes

chore: address review comments on Makefile, Readme and dockerFile

b1e0a68

Bidon15 requested a review from rootulp July 18, 2023 11:29

MSevey requested a review from a team July 18, 2023 11:29

This was referenced Jul 18, 2023

docker: Prettify comment sections in Dockerfile and Dockerfile.ephemeral. #2117

Closed

docker/doc: Document Dockerfile and Dockerfile.ephemeral in the docker/README.md #2118

Closed

Apply suggestions from code review

6f5204b

Co-authored-by: Rootul P <[email protected]>

rootulp approved these changes Jul 18, 2023

View reviewed changes

evan-forbes approved these changes Jul 18, 2023

View reviewed changes

MSevey reviewed Jul 18, 2023

View reviewed changes

evan-forbes approved these changes Jul 24, 2023

View reviewed changes

rootulp merged commit 7494cc7 into celestiaorg:main Jul 24, 2023
20 checks passed

Bidon15 deleted the txsim-cmd branch August 4, 2023 08:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: make a txsim binary build from Makefile and dockerfile #2052

feat: make a txsim binary build from Makefile and dockerfile #2052

Bidon15 commented Jul 10, 2023 •

edited

Loading

codecov-commenter commented Jul 10, 2023 •

edited

Loading

tty47 left a comment

rootulp Jul 17, 2023

Bidon15 Jul 18, 2023

rootulp Jul 17, 2023

Bidon15 Jul 18, 2023

rootulp Jul 17, 2023

Bidon15 Jul 18, 2023 •

edited

Loading

rootulp Jul 18, 2023

rootulp left a comment

rootulp Jul 18, 2023

evan-forbes left a comment

MSevey Jul 18, 2023

MSevey Jul 18, 2023

evan-forbes left a comment

evan-forbes commented Jul 24, 2023

		@@ -0,0 +1,122 @@
		# Celestia `txsim` Docker Image Usage Guide

		The `txsim` binary is a tool that can be used to simulate transactions on the Celestia network. It can be used to test the performance of the Celestia network.

	docker run -it -v ${HOME}/.celestia-app:/home/celestia ghcr.io/celestiaorg/txsim -k 0 -r http://consensus-validator-robusta-rc6.celestia-robusta.com:26657,http://consensus-full-robusta-rc6.celestia-robusta.com:26657 -g consensus-validator-robusta-rc6.celestia-robusta.com:9090 -t 10s -b 10 -d 100 -e 10
	docker run --interactive --tty --volume ${HOME}/.celestia-app:/home/celestia ghcr.io/celestiaorg/txsim -k 0 -r http://consensus-validator-robusta-rc6.celestia-robusta.com:26657,http://consensus-full-robusta-rc6.celestia-robusta.com:26657 -g consensus-validator-robusta-rc6.celestia-robusta.com:9090 -t 10s -b 10 -d 100 -e 10

	docker build -t ghcr.io/celestiaorg/txsim -f docker/Dockerfile_txsim .
	docker build --tag ghcr.io/celestiaorg/txsim --file docker/Dockerfile_txsim .

feat: make a txsim binary build from Makefile and dockerfile #2052

feat: make a txsim binary build from Makefile and dockerfile #2052

Conversation

Bidon15 commented Jul 10, 2023 • edited Loading

codecov-commenter commented Jul 10, 2023 • edited Loading

Codecov Report

tty47 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Bidon15 Jul 18, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rootulp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

evan-forbes left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

evan-forbes left a comment

Choose a reason for hiding this comment

evan-forbes commented Jul 24, 2023

Bidon15 commented Jul 10, 2023 •

edited

Loading

codecov-commenter commented Jul 10, 2023 •

edited

Loading

Bidon15 Jul 18, 2023 •

edited

Loading