Benchmark

[robertDurst]

As we move to actually making this thing run a little faster (#7 and #8), it might be interesting to have some benchmark data.

[grempe]

Benchmarking data point.

I generated a vanity address for a 7 character prefix (all uppercase alpha, no numbers).

It took 316858.619223915 seconds (88 hours) on a 96 core AWS c5.24xlarge instance.

I suppose that is 8,448 'core hours' of computation.

Significantly higher than the "3 hours" estimate in the README chart (which also doesn't specify computing power assumed for those numbers). YMMV

[robertDurst]

Thank you very much for doing this!

I believe I borrowed this diagram from some website when I first created this repo. Not only does it not specify computing power assumed for those numbers I am not even 100% sure it's based on the same address formatting.

[robertDurst]

Going to play around with this some this week and derive a chart based on actual observed data.

[grempe]

One other thing to note that surprised me. The resulting address, based on choosing a prefix (let’s call it “MYWORDS”) was “GCMYWORDS...” and I didn’t expect the leading “C” to be in there.

Any thoughts? I didn’t see anything in the “strkey“ format to account for it.

Thanks

[robertDurst]

Ah, I see what you mean. Yes, it is a bit weird, however it is a side effect of the way stellar addresses are generated. Specifically, it is the result of adding a version byte:

 const VERSION_BYTE_ACCOUNT_ID: u8 = 6 << 3;

Once this is base32 encoded, it turns out the first two characters are limited to:

• GA
• GB
• GC
• GD

If you create a few accounts here in the Stellar Laboratory you can see what I mean.

[robertDurst]

Code actually being benchmarked now with criterion. Only can get access to 32 cores on DigitalOcean, so probably won't test super long prefixes/postfixes.

[robertDurst]

Updated with first go at benchmarking.

[grempe]

If useful, if you provide in the README some instructions for running the benchmark I’ll be happy to run it on a larger machine. And perhaps up to six letters in a run? I think I can get up to 128 cores on AWS.

FYI, 5 letter prefixes on that machine took about 3-5m each across about 5 runs.

I didn’t test six letters, I jumped right to seven as shown in my original post.

[robertDurst]

5 letter prefixes on that machine took about 3-5m each across about 5 runs.

Sweet!

If useful, if you provide in the README some instructions for running the benchmark I’ll be happy to run it on a larger machine. And perhaps up to six letters in a run?

That would be super helpful! Give me a second, I'll add a readme & make the benchmark configurable.

[robertDurst]

@grempe ok, I believe this should suffice: https://github.com/robertDurst/stellar-vanity-address-generator#how-can-i-benchmark

Please note the parameters in configurations.

[grempe]

OK, I ran the benchmark. I had to make some modifications to allow it to run in a reasonable time (a 96 core machine is not cheap!). I hope this is useful.

Modifications:

I changed the number of benchmark iterations to 10 and removed the six character example (it would have taken approximately a month to run!!!). The runtime for this benchmark was about a day. Actually, I'm not quite sure why it took so long to run as the most intensive part (5 char) should have been done with 10 runs in much less than that (~500s * 10 = 1.38h). Is there an inner loop to this benchmark that makes it run more iterations and run much longer?

Server Info:

AWS c5.metal (358 ECUs, 96 vCPUs, 3.6 GHz, 2nd Gen Intel Xeon Platinum 8275CL, 192 GiB memory, EBS only)

25 Gigabit network

Ubuntu Server 18.04 LTS (HVM), SSD Volume Type - ami-085925f297f89fce1

Installation Steps:

sudo apt-get update
sudo apt-get install -y gcc build-essential gnuplot

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

source $HOME/.cargo/env

git clone https://github.com/robertDurst/stellar-vanity-address-generator.git

cd stellar-vanity-address-generator

cargo bench

Benchmark Run STDOUT:

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out


running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

Benchmarking prefix/one
Benchmarking prefix/one: Warming up for 3.0000 s
Benchmarking prefix/one: Collecting 10 samples in estimated 5.1832 s (770 iterations)
Benchmarking prefix/one: Analyzing
prefix/one              time:   [6.5281 ms 6.6769 ms 6.8162 ms]
Benchmarking prefix/two
Benchmarking prefix/two: Warming up for 3.0000 s
Benchmarking prefix/two: Collecting 10 samples in estimated 6.0361 s (165 iterations)
Benchmarking prefix/two: Analyzing
prefix/two              time:   [37.183 ms 38.213 ms 39.238 ms]
Benchmarking prefix/three
Benchmarking prefix/three: Warming up for 3.0000 s

Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 29.5s.
Benchmarking prefix/three: Collecting 10 samples in estimated 29.521 s (55 iterations)
Benchmarking prefix/three: Analyzing
prefix/three            time:   [629.38 ms 889.87 ms 1.0218 s]
Benchmarking prefix/four
Benchmarking prefix/four: Warming up for 3.0000 s

Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 202.6s.
Benchmarking prefix/four: Collecting 10 samples in estimated 202.63 s (55 iterations)
Benchmarking prefix/four: Analyzing
prefix/four             time:   [7.6370 s 9.7025 s 12.201 s]
Benchmarking prefix/five
Benchmarking prefix/five: Warming up for 3.0000 s

Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 36949.8s.
Benchmarking prefix/five: Collecting 10 samples in estimated  36950 s (55 iterations)
Benchmarking prefix/five: Analyzing
prefix/five             time:   [209.23 s 448.19 s 606.54 s]

I installed gnuplot on the server so you'd get the pretty HTML reports as well. I've tar'ed up the whole output results for you to download here.

https://svag-benchmark.s3.amazonaws.com/svag-target.tar.gz

As previously reported, I think a six character run prob takes the better part of a day to run. And my previous report about 7 characters taking 3 days should be consistent with these results as it was on the same number of cores (96).

I think an 8 character run would start to get very expensive.

Please download the results for your own use as I'll delete them after a little while.

Cheers.

[grempe]

Another thought. Should the number of CPU's consumed by the benchmark be less than the total CPU's on the system? Otherwise every CPU gets pegged at 100% which doesn't allow room for the system and the benchmark runner to do work?

[robertDurst]

Sweet, thanks a lot for doing this - downloaded the plots.

Should the number of CPU's consumed by the benchmark be less than the total CPU's on the system? Otherwise every CPU gets pegged at 100% which doesn't allow room for the system and the benchmark runner to do work?

This is a good point. I am going to play around with this, this weekend.

[robertDurst]

I'll be honest, I'm not quite 100% sure how to test the difference of saving a CPU for the system and benchmark. That being said, this makes sense and I added -1 to the get max cpus in the benchmark.

Also added your results to the README. Thanks again for that! Don't have access to that many CPU's 😅

[grempe]

The results posted are good, but they are for pretty small length strings at 5 chars max. The jump to 7 chars, you’ll remember from my earliest posting ran for 88 hours. It’s kind of a flaw in the benchmark approach as a bench run for 6 or 7 chars might take weeks to run and cost thousands. This will prob never be run. 8 chars may take much much longer. Months? Years?

Some text around this in the readme might make sense to set expectations.

[robertDurst]

Good point. Will update this.