BUIP126: Planet-on-a-LAN stress test model network
Proposer: jtoomim
Submitted on: 2019-05-13
Status: passed
I'd like to set up a LAN-based test apparatus for simulating a planet-wide network of nodes, using the Linux netem module to add network delays, packet loss, and bandwidth caps as needed. This will provide a controlled environment in which developers can test out new code or new configurations quickly and efficiently, and rapidly collect and collate performance data.
The suggested nature of this network will be a server rack full of used servers, all interconnected via Gigabit Ethernet, operated in my datacenter in Moses Lake, WA on a separate LAN with a dedicated 100 Mbps connection to the internet. These servers will likely cost around $800 each after adding SSDs and HDDs, and will come with 16 to 40 CPU cores and roundabout 128 GB RAM. Each physical machine can run multiple nodes (either in separate VMs or just on different ports of the same machine). For around $8,000, we can set up 10 servers and get around 40 to 100 network nodes.
Operating costs for a 10 machine setup will be about $60/month for the 100 Mbps internet connection plus around $80/month for electricity costs, plus some undetermined amount for labor from my employees. In contrast, renting 10 dedicated servers with this kind of HW specs would cost around $2,000/month if we ordered it from standard cloud hosting providers. Owning will be cheaper than renting if this network is in operation for more than 4 months.
I expect to be able to keep this rig in operation at least until April 1st, 2020, at which time it will likely need to be relocated.
These machines can either be set up with one static global IP address per machine or behind NAT with forwarded ports for SSH and other services. I personally prefer the NAT/port forward concept, as I expect that will be cheaper, more scalable, and more easily relocated.
Experiments will probably usually use regtest mode. Use in testnet mode as part of an actual global network can also be done with machines in this network, but the 100 Mbps outbound pipe may become a bottleneck.
For ssh, non-simulation admin tasks, and performance data collection, we can set up a separate LAN (using the secondary Ethernet ports) without any artificial latency or packet loss.
This test setup is intended to be made available by all developers in the BCH community, not just BU developers. Experiments using a heterogenous mixture of node implementations will be allowed, as will experiments using a homogenous set of nodes that are not BU (e.g. bchd alone, or ABC alone). I expect BU's developers will make heavier use of it, though, as BU's developers tend to be more scaling-focused than the other implementations.
I will probably end up just buying this gear outright and setting it up with or without BU's financial support. However, if BU's membership wishes to reimburse me for the hardware costs by passing this BUIP, I will accept that support.
I am also interested in hearing how big our members think we should make this network. 10 machines? 5? 50? Small networks will outperform big ones, and many performance problems might not be apparent unless we have high per-node peer counts or high hop counts for total tx and block propagation paths. But also, mo nodes mo moneh. So. How much?
Discussion of this project can happen in https://t.me/BCH_stress_testnet.