-
Notifications
You must be signed in to change notification settings - Fork 55
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
be823c1
commit 964a349
Showing
2 changed files
with
183 additions
and
38 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,26 +1,94 @@ | ||
| ## Changelog | ||
|
|
||
| ### Beta 0.1.0 (22/03/2022) | ||
|
|
||
| ### v0.3.0 (03/11/2024) | ||
| **New Features:** | ||
| - Support Data Parallelism | ||
| - Support 1D Tensor Parallelism | ||
| - Support Pipeline Parallelism | ||
| - Unified distributed Layers for both single-GPU and multi-GPU training | ||
| - `LazyConfig` system for more flexible syntax and no predefined structures | ||
| - Easy-to-use trainer and engine | ||
| - Support both CV and NLP data processing | ||
| - Mixed Precision Training | ||
| - Activation Checkpointing | ||
| - Gradient Accumulation | ||
| - Gradient Clipping | ||
| - Zero Redundancy Optimizer (ZeRO) | ||
| - Support mock transformers, see [Mock transformers](https://github.com/Oneflow-Inc/libai/tree/main/projects/mock_transformers#readme) | ||
| - Support lm-evaluation-harness for model evaluation | ||
| - User Experience Optimization | ||
|
|
||
| **New Supported Models:** | ||
| - These models are natively supported by libai | ||
| <table class="docutils"> | ||
| <tbody> | ||
| <tr> | ||
| <th width="130"> Models </th> | ||
| <th valign="bottom" align="center" width="140"> 2D(tp+pp) Inference</th> | ||
| <th valign="bottom" align="center" width="140"> 3D Parallel Training </th> | ||
| </tr> | ||
| <tr> | ||
| <td align="center"><a href="https://github.com/Oneflow-Inc/libai/tree/main/projects/BLOOM"> <b> BLOOM </b> </td> | ||
| <td align="center">✔</td> | ||
| <td align="center">-</td> | ||
| </tr> | ||
| <tr> | ||
| <td align="center"><a href="https://github.com/Oneflow-Inc/libai/tree/main/projects/ChatGLM"> <b> ChatGLM </b> </td> | ||
| <td align="center">✔</td> | ||
| <td align="center">✔</td> | ||
| </tr> | ||
| <tr> | ||
| <td align="center"><a href="https://github.com/Oneflow-Inc/libai/tree/main/projects/Couplets"> <b> Couplets </b> </td> | ||
| <td align="center">✔</td> | ||
| <td align="center">✔</td> | ||
| </tr> | ||
| <tr> | ||
| <td align="center"><a href="https://github.com/Oneflow-Inc/libai/tree/main/projects/DALLE2"> <b> DALLE2 </b> </td> | ||
| <td align="center">✔</td> | ||
| <td align="center">-</td> | ||
| </tr> | ||
| <tr> | ||
| <td align="center"><a href="https://github.com/Oneflow-Inc/libai/tree/main/projects/Llama"> <b> Llama2 </b> </td> | ||
| <td align="center">✔</td> | ||
| <td align="center">✔</td> | ||
| </tr> | ||
| <tr> | ||
| <td align="center"><a href="https://github.com/Oneflow-Inc/libai/tree/main/projects/MAE"> <b> MAE </b> </td> | ||
| <td align="center">✔</td> | ||
| <td align="center">✔</td> | ||
| </tr> | ||
| <tr> | ||
| <td align="center"><a href="https://github.com/Oneflow-Inc/libai/tree/main/projects/Stable_Diffusion"> <b> Stable_Diffusion </b> </td> | ||
| <td align="center">-</td> | ||
| <td align="center">-</td> | ||
| </tr> | ||
| </tbody> | ||
| </table> | ||
|
|
||
| **Supported Models:** | ||
| - Support 3D parallel [BERT](https://arxiv.org/abs/1810.04805) model | ||
| - Support 3D parallel [GPT-2](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) model | ||
| - Support 3D parallel [T5](https://arxiv.org/abs/1910.10683) model | ||
| - Support 3D parallel [Vision Transformer](https://arxiv.org/abs/2010.11929) | ||
| - Support Data parallel [Swin Transformer](https://arxiv.org/abs/2103.14030) model | ||
| - Support finetune task in [QQP project](/projects/QQP/) | ||
| - Support text classification task in [text classification project](/projects/text_classification/) | ||
| **New Mock Models:** | ||
| - These models are extended and implemented by libai through mocking transformers. | ||
| <table class="docutils"> | ||
| <tbody> | ||
| <tr> | ||
| <th width="130"> Models </th> | ||
| <th valign="bottom" align="center" width="140">Tensor Parallel</th> | ||
| <th valign="bottom" align="center" width="150">Pipeline Parallel</th> | ||
| </tr> | ||
| <tr> | ||
| <td align="center"><a href="https://huggingface.co/docs/transformers/v4.26.1/en/model_doc/bloom#overview"> <b> BLOOM </b> </td> | ||
| <td align="center">✔</td> | ||
| <td align="center">-</td> | ||
| </tr> | ||
| <tr> | ||
| <td align="center"><a href="https://github.com/openai/gpt-2/blob/master/model_card.md"> <b> GPT2 </b> </td> | ||
| <td align="center">✔</td> | ||
| <td align="center">-</td> | ||
| </tr> | ||
| <tr> | ||
| <td align="center"><a href="https://huggingface.co/docs/transformers/v4.28.0/en/model_doc/llama#overview"> <b> LLAMA </b> </td> | ||
| <td align="center">✔</td> | ||
| <td align="center">-</td> | ||
| </tr> | ||
| <tr> | ||
| <td align="center"><a href="https://huggingface.co/docs/transformers/main/en/model_doc/llama2"> <b> LLAMA2 </b> </td> | ||
| <td align="center">✔</td> | ||
| <td align="center">-</td> | ||
| </tr> | ||
| <tr> | ||
| <td align="center"><a href="https://huggingface.co/baichuan-inc/Baichuan-7B"> <b> Baichuan </b> </td> | ||
| <td align="center">✔</td> | ||
| <td align="center">-</td> | ||
| </tr> | ||
| <tr> | ||
| <td align="center"><a href="https://huggingface.co/docs/transformers/v4.26.1/en/model_doc/opt#overview"> <b> OPT </b> </td> | ||
| <td align="center">✔</td> | ||
| <td align="center">-</td> | ||
| </tr> | ||
| </tbody> | ||
| </table> |