What are we looking for in a benchmarking tool?

[SWilson4]

As discussed in the 2024/04/09 OQS developers call, we (well, I) would like to revamp this subproject so that we can collect useful and current benchmarking data. Before diving in, we should assess what we hope to get out of the work.

Here are some preliminary questions that come to mind:

How do we intend to use benchmarking data?

For example: Do we want to inform "data-driven" decisions about algorithm use? Test for performance regressions after algorithm updates? Provide data for researchers to use in tables in their papers? The answers to this will naturally inform the second question:

What data do we want to collect?

The current (old?) benchmarking code defines five levels for data collection, three of which are/were implemented. Does this still reflect the data we want to capture?

Where do we want this to run?

We had been using an ad-hoc collection of AWS VMs and Douglas's M1 Macbook. Do we want to add more platforms (e.g., s390x)? Should we add some performance-testing guarantees to platform support documentation? Do we want to keep running everything in Docker containers, or should we run on the host OS directly?

Where do we want the code to live?

New repo? New name? Within liboqs? etc.

Let the discussion begin...

[SWilson4]

I'll get things started with my thoughts on the above questions.

The main use cases I see for benchmarking data are

1. evaluating algorithmic optimizations,
2. comparing algorithm candidates, and
3. regression testing.

I think (2) is perhaps the most important to the crypto community at large, especially as we bring on board more candidates from the NIST signature on-ramp. I think (3) would be a good addition to our automated release testing. (We did this manually, sometimes, but it's not clear to me how consistently.)

On that front, I think we ought to keep collecting library-level (keygen / encaps / sign / etc) data as well as application- and network-level data. I imagine that TLS handshake performance data in particular will be valuable for evaluating new signature algorithms.

Somewhat complicated infrastructure setup has given us problems in the past, and I think we have a good chance to correct this with a rewrite. IMO, we should track setup/trigger scripts in GitHub whenever possible. Ideally, I think we would run on a CI platform, but I don't think that we would get reliable data that way. (Worth investigating?) For a start, I think we should collect data on x64/arm64 and add support for other platforms by popular demand/contribution.

To sum it up, I would say that the previous profiling code did a good job of capturing most of the data I would consider valuable; the setup was simply difficult to maintain and debug when things went askew. This is by no means an affront to @baentsch, who (as I understand it) wrote most of the previous code on his own. It seems to me that we now have more resources to throw at the problem and hence make improvements.

As for where to put this... I'm all for a fresh start in a new repo, which links back to this one as appropriate for documentation. I also submit that "profiling" is a word with connotations best avoided and suggest open-quantum-safe/benchmarking as a new name.

Tagging @open-quantum-safe/tsc for input, as this hits both "approving project or system proposals (including, but not limited to, incubation, deprecation, and changes to a sub-project’s scope)" and "organizing sub-projects and removing sub-projects". Also @open-quantum-safe/core.

[bhess]

Thank you @SWilson4 for taking that on! I think both use cases, regression testing and evaluating/comparing algorithm candidates are very valuable.
For testing, another useful thing could be to include the common code in the benchmarking since many algorithms highly depend on, e.g., Keccak performance.
For evaluating/comparing algorithms, another useful metric could be memory/stack usage.
Then more "real-world" scenario such as level 4/5 in the current 'profiling' project would be nice as well, although probably tricker in the setup. Perhaps this could be done in a "less automated" way. I'm interested to contribute as well (not sure about the timing though yet).