Add utility functions to obtain elapsed time and communication cost

[MarcT0K]

Provides two utility functions to compute the elapsed time and communication cost since the start of the MPyC runtime.

[codecov-commenter]

Codecov Report

Patch coverage: 100.00% and no project coverage change.

Comparison is base (aa71585) 92.54% compared to head (90534fd) 92.55%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files

@@           Coverage Diff           @@
##           master      #50   +/-   ##
=======================================
  Coverage   92.54%   92.55%           
=======================================
  Files          16       16           
  Lines        8451     8458    +7     
=======================================
+ Hits         7821     7828    +7     
  Misses        630      630           

Impacted Files	Coverage Δ
mpyc/runtime.py	90.28% <100.00%> (+0.02%)	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[lschoe]

Thanks for the PR. I'm a bit hesitant with extending the API in this direction, so let's keep this issue open for some time. Until after adding some basic support for generating more elaborate performance statistics inside MPyC. (Using a decorator, such that runs without stats are not degraded.)

For now, other users interested in the elapsed time and number of bytes could use a simple function like you are doing. Do you use other functions like these? Or do you need more functionality in this direction?

[MarcT0K]

I'm a bit hesitant with extending the API in this direction, so let's keep this issue open for some time

I totally understand. I developed (for my own experiments) these simple utility functions so I felt it was easier to submit a PR instead of an issue to discuss about them.

Do you use other functions like these? Or do you need more functionality in this direction?

For now, I do not use any other functions. Runtime and communication cost are the two traditional metrics shown in MPC papers and I am not aware of any other metrics asked in MPC literature. A somewhat related request could be to simulate latency in the protocol (and to set a bandwidth limit). With such a feature, we may reproduce more easily the "LAN" and "WAN" setups. I know it is not exactly about performance monitoring but it is a subject somehow related.

By the way, I am willing to give a hand in these changes if necessary. I submitted two first "simple" PRs to get used to MPyC standards.

[lschoe]

To simulate LAN/WAN settings I would rather keep that outside the framework and use other tools for that. Unless it can be done in a very nonintrusive way.

But there is room for more precise performance metrics. It's not just runtime and communication cost. For example, in lots of protocols the amount of local computation per party is not negligible. It's therefore interesting to see how much time is spent on this, and to what extent the local work can be parallelized (or not); or can be vectorized whatever that means exactly. And how much of this can be done in preprocessing.

For communication cost that can be broken down into communication complexity (counting number of bytes sent), but also round complexity, as determined by the longest path in the computation graph (or, circuit), Maybe some form of message complexity too, counting how many exchanges there are at the level of mpc._send_message() for example.

[MarcT0K]

To simulate LAN/WAN settings I would rather keep that outside the framework and use other tools for that. Unless it can be done in a very nonintrusive way.

I understand your point. I guess some (async) sleep calls could be performed to add some latency but for the bandwidth it might get a little bit more complex.

Regarding the additional metrics, I agree that your ideas are interesting. Currently, I deduce these metrics indirectly from the observation of the runtime and the communication cost but having access to such metrics may help debugging the code. The round complexity measurement is particularly interesting to check the correctness of an implementation.