[CI] Add load-testing

[okdas]

Objective

We need to add optional load testing to DevNet environments.

Origin Document

• https://discord.com/channels/824324475256438814/1138895490331705354/1256614409975894028
• https://discord.com/channels/824324475256438814/1138895490331705354/1266483844618518538

Goals

• Ensure the stress code remains functional after architectural changes (such as depinject module additions/modifications)
• Investigate potential data points obtainable from running stress tests on DevNet as part of CI

Deliverables

• CI modification to execute stress tests when a relevant GitHub label is attached to the PR

Non-goals / Non-deliverables

• Regular stress tests for each PR; we aim to run them conditionally
• Gain visibility into how the suite reacts to different code changes.
  • This may not be practical to implement with the current DevNet architecture, and more historical data is needed to determine the next steps for implementation. This can be addressed later once sufficient historical data is gathered.

General deliverables

• Comments: Add/update TODOs and comments alongside the source code so it is easier to follow.
• Testing: Add new tests (unit and/or E2E) to the test suite.
• Makefile: Add new targets to the Makefile to make the new functionality easier to use.
• Documentation: Update architectural or development READMEs; use mermaid diagrams where appropriate.

---

Creator: @okdas

[okdas]

I don't think we're going to add this to CI in the following iterations. I moved the ticket to look into adding load test to CI in November.