-
Notifications
You must be signed in to change notification settings - Fork 17
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Plugin support #263
base: main
Are you sure you want to change the base?
Plugin support #263
Conversation
8e3e909
to
17f8388
Compare
17f8388
to
e5841d2
Compare
ec87e78
to
07fc86f
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For existing prologue we use real NCCL run. In your examples it seems that we are switching to some predefined commands.
- How are we going to generate it?
- Will that cover our needs? cc @srivatsankrishnan
I do have some code related notes, but let's leave it for later discussion.
Yes, it is one of the main design choices that we need to make. |
Can we rely on existing mechanisms? Each plugin will be defined as a regular Test TOML, meaning we can generate a CLI for it for a particular system. This is what we do now and it seems to cover all our needs for this feature. |
7594c19
to
852fee8
Compare
Summary
This PR introduces plugins to CloudAI. Plugins are tests that run either before or after each test in a test scenario. They are defined globally within a test scenario and are automatically executed for each test. There are two types of plugins: prologues and epilogues. Prologues run before the tests, while epilogues are executed after the tests. Multiple prologues and epilogues can be specified in each scenario.
An example of how plugins are defined within a test scenario:
You can see the prologue and epilogue fields. These are used to look up the corresponding plugin file. A plugin file is a separate test scenario file as shown below:
If any of the tests in the prologue fail, the main test or the epilogue tests will not run. In other words, the main test and epilogue run conditionally when the prologue is successful. The tests in plugins have time limits, just as tests in the main scenario do. Output files should be stored in the output directory, in a subdirectory called "prologue" or "epilogue," following a proper directory hierarchy. Plugins are not supported for NeMo 1.0 (NeMo launcher).
Note
Test Plan
2.1 Success
2.2 Failure