Plugin support

[TaekyungHeo]

Summary

This PR introduces plugins to CloudAI. Plugins are tests that run either before or after each test in a test scenario. They are defined globally within a test scenario and are automatically executed for each test. There are two types of plugins: prologues and epilogues. Prologues run before the tests, while epilogues are executed after the tests. Multiple prologues and epilogues can be specified in each scenario.

An example of how plugins are defined within a test scenario:

name = "nccl-test"

prologue = "nccl_test_prologue"
epilogue = "nccl_test_epilogue"

[[Tests]]
id = "Tests.1"
test_name = "nccl_test_all_reduce"
num_nodes = "2"
time_limit = "00:20:00"

[[Tests]]
id = "Tests.2"
test_name = "nccl_test_all_gather"
num_nodes = "2"
time_limit = "00:20:00"
  [[Tests.dependencies]]
  type = "start_post_comp"
  id = "Tests.1"

You can see the prologue and epilogue fields. These are used to look up the corresponding plugin file. A plugin file is a separate test scenario file as shown below:

name = "nccl_test_prologue"

[[Tests]]
id = "Tests.1"
test_name = "nccl_test_all_reduce"
time_limit = "00:20:00"

If any of the tests in the prologue fail, the main test or the epilogue tests will not run. In other words, the main test and epilogue run conditionally when the prologue is successful. The tests in plugins have time limits, just as tests in the main scenario do. Output files should be stored in the output directory, in a subdirectory called "prologue" or "epilogue," following a proper directory hierarchy. Plugins are not supported for NeMo 1.0 (NeMo launcher).

Note

• Previous Design Document: https://docs.google.com/document/d/1fqD_hBXmj0ikXX91iAVGvCLZk2tZdUKqK08TZdT7E0g/edit#heading=h.5cwfklh0ybde
• TODO: Document how it works and share it.
• Term - plugin / run wrapper / pre-flight, post-flight
• Idea
  • We may need to generate reports from plugins.
  • We may need to consider the performance impact of plugins.
  • Dependencies are not implemented for now.

Test Plan

1. CI passes
2. Manual run
   2.1 Success

$ cloudai run --system-config ~/cloudaix-main/conf/common/system/israel_1.toml --tests-dir conf/common/test --test-scenario conf/common/test_scenario/nccl_tes
t.toml
/.autodirect/mswg2/E2E/theo/venv/lib/python3.10/site-packages/requests/__init__.py:102: RequestsDependencyWarning: urllib3 (1.26.19) or chardet (5.2.0)/charset_normalizer (2.0.12) doesn't match a supported ver
sion!
  warnings.warn("urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn't match a supported "                                                                                                                
[INFO] System Name: Israel-1                                                                            
[INFO] Scheduler: slurm                                                                                 
[INFO] Test Scenario Name: nccl-test                                                                    
[INFO] Checking if test templates are installed.                                                        
[INFO] Test Scenario: nccl-test                                                                         

Section Name: Tests.1                                                                                   
  Test Name: nccl_test_all_reduce                                                                       
  Description: all_reduce                                                                               
  No dependencies                                                                                       
[INFO] Initializing Runner [RUN] mode                                                                   
[INFO] Creating SlurmRunner                                                                             
[INFO] Starting test scenario execution.                                                                
[INFO] Starting test: Tests.1                                                                           
[INFO] Running test: Tests.1                                                                            
[INFO] Executing command for test Tests.1: sbatch /auto/e2e/israel1/workload_results/nccl-test_2024-10-25_22-05-46/Tests.1/0/cloudai_sbatch_script.sh
[INFO] Job completed: Tests.1
[INFO] All test scenario results stored at: /auto/e2e/israel1/workload_results/nccl-test_2024-10-25_22-05-46
[INFO] All test scenario execution attempts are complete. Please review the 'debug.log' file to confirm successful completion or to identify any issues.

$ cd /auto/e2e/israel1/workload_results/nccl-test_2024-10-25_22-05-46/Tests.1/0
$ ls
cloudai_sbatch_script.sh  epilogue  prologue  stderr.txt  stdout.txt

$ ls prologue/nccl_test_all_reduce/
stderr.txt  stdout.txt

$ ls epilogue/nccl_test_all_gather/
stderr.txt  stdout.txt

2.2 Failure

$ cloudai run --system-config ~/cloudaix-main/conf/common/system/israel_1.toml --tests-dir conf/common/test --test-scenario conf/common/test_scenario/nccl_test.toml
/.autodirect/mswg2/E2E/theo/venv/lib/python3.10/site-packages/requests/__init__.py:102: RequestsDependencyWarning: urllib3 (1.26.19) or chardet (5.2.0)/charset_normalizer (2.0.12) doesn't match a supported version!
  warnings.warn("urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn't match a supported "
[INFO] System Name: Israel-1
[INFO] Scheduler: slurm
[INFO] Test Scenario Name: nccl-test
[INFO] Checking if test templates are installed.
[INFO] Test Scenario: nccl-test

Section Name: Tests.1
  Test Name: nccl_test_all_reduce
  Description: all_reduce
  No dependencies
[INFO] Initializing Runner [RUN] mode
[INFO] Creating SlurmRunner
[INFO] Starting test scenario execution.
[INFO] Starting test: Tests.1
[INFO] Running test: Tests.1
[INFO] Executing command for test Tests.1: sbatch /auto/e2e/israel1/workload_results/nccl-test_2024-10-25_22-16-25/Tests.1/0/cloudai_sbatch_script.sh
[ERROR] Job 383928 for test Tests.1 failed: Missing success indicators in /auto/e2e/israel1/workload_results/nccl-test_2024-10-25_22-16-25/Tests.1/0/stdout.txt: '# Out of bounds values', '# Avg bus bandwidth'. These keywords are expected to be present in stdout.txt, usually towards the end of the file. Please review the NCCL test output and errors in the file. Ensure the NCCL test ran to completion. You can run the generated sbatch script manually and check if /auto/e2e/israel1/workload_results/nccl-test_2024-10-25_22-16-25/Tests.1/0/stdout.txt is created and contains the expected keywords. If the issue persists, contact the system administrator.
[INFO] Terminating all jobs...
[INFO] All jobs have been killed.

[amaslenn]

For existing prologue we use real NCCL run. In your examples it seems that we are switching to some predefined commands.

1. How are we going to generate it?
2. Will that cover our needs? cc @srivatsankrishnan

I do have some code related notes, but let's leave it for later discussion.

[TaekyungHeo]

@amaslenn

How are we going to generate it?

Yes, it is one of the main design choices that we need to make.

[amaslenn]

Yes, it is one of the main design choices that we need to make.

Can we rely on existing mechanisms? Each plugin will be defined as a regular Test TOML, meaning we can generate a CLI for it for a particular system. This is what we do now and it seems to cover all our needs for this feature.