🇨🇳中文 | 🌐English | 📖文档/Docs | ❓提问/Issues | 💬讨论/Discussions
Large Language Models (LLM), represented by ChatGPT and GPT-4, have sparked a new wave of research in the field of natural language processing, demonstrating capabilities of Artificial General Intelligence (AGI) and attracting widespread attention from the industry. However, the expensive training and deployment of large language models have posed certain obstacles to building transparent and open academic research.
To promote open research of large models in the Chinese NLP community, this project has open-sourced the Chinese LLaMA model and the Alpaca large model with instruction fine-tuning. These models expand the Chinese vocabulary based on the original LLaMA and use Chinese data for secondary pre-training, further enhancing Chinese basic semantic understanding. Additionally, the project uses Chinese instruction data for fine-tuning on the basis of the Chinese LLaMA, significantly improving the model's understanding and execution of instructions. Please refer to our technical report for further details (Cui, Yang, and Yao, 2023).
Main contents of this project:
- 🚀 Extended Chinese vocabulary on top of original LLaMA with significant encode/decode efficiency
- 🚀 Open-sourced the Chinese LLaMA (general purpose) and Alpaca (instruction-tuned)
- 🚀 Open-sourced the pre-training and instruction finetuning (SFT) scripts for further tuning on user's data
- 🚀 Quickly deploy and experience the quantized version of the large model on CPU/GPU of your laptop (personal PC)
- 🚀 Support 🤗transformers, llama.cpp, text-generation-webui, LlamaChat, LangChain, , privateGPT, etc.
- Released versions: 7B (basic, Plus), 13B (basic, Plus) , 33B (basic)
💡 The following image shows the actual experience effect of the 7B version model after local deployment (animation unaccelerated, tested on Apple M1 Max).
Multi-modal VLE | Chinese MiniRBT | Chinese LERT | Chinese-English PERT | Chinese MacBERT | Chinese ELECTRA | Chinese XLNet | Chinese BERT | Knowledge distillation tool TextBrewer | Model pruning tool TextPruner
[June 16, 2023] Release v4.1: New technical report, add C-Eval inference script, add low-resource model merging script, etc.
[June 8, 2023] Release v4.0: LLaMA/Alpaca 33B versions are available. We also add privateGPT demo, C-Eval results, etc.
[May 16, 2023] Release v3.2: Add SFT scripts, LangChain supports, Gradio-based web demo, etc.
[May 10, 2023] Release v3.1: LLaMA/Alpaca Plus 13B versions are available, more training data used than basic ones.
[Apr 28, 2023] Release v3.0: LLaMA/Alpaca Plus versions are available, more training data used than basic ones.
Previous News
[Apr 18, 2023] Release v2.2: Add LlamaChat support (macOS UI), tokenizer merging scripts, documentations are migrated to GitHub Wiki. Refer to Release Note
[Apr 13, 2023] Release v2.1: Add HuggingFace-transformers and text-generation-webui interfances. Refer to Release Note
[Apr 07, 2023] Release v2.0: Release 13B versions of Chinese LLaMA and Alpaca model. Main upgrades: stronger factuality, better performance on QA, translation and more. Refer to Release Note
[Mar 31, 2023] Release v1.1, major updates: simplification of model merging steps, addition of instruction data crawling script, and important notes about the new version of llama.cpp. See Release Note.
[Mar 28, 2023] Open-sourcing Chinese LLaMA and Alpaca, currently offering the 7B version for download and experience
| Chapter | Description |
|---|---|
| Download | Download links for Chinese LLaMA and Alpaca |
| Model Reconstruction | (Important) Explains how to merge downloaded LoRA models with the original LLaMA |
| Quick Deployment | Steps for quantize and deploy LLMs on personal computers |
| Example Results | Examples of the system output |
| Training Details | Introduces the training details of Chinese LLaMA and Alpaca |
| FAQ | Replies to some common questions |
| Limitations | Limitations of the models involved in this project |
The official LLaMA models released by Facebook prohibit commercial use, and the official model weights have not been open-sourced (although there are many third-party download links available online). In order to comply with the relevant licenses, it is currently not possible to release the complete model weights. We appreciate your understanding. After Facebook fully opens up the model weights, this project will update its policies accordingly. What is released here are the LoRA weights, which can be seen as a "patch" for the original LLaMA model, and the complete weights can be obtained by merging the two.
The following table provides a basic comparison of the Chinese LLaMA and Alpaca models, as well as recommended usage scenarios (including, but not limited to).
💡 Plus versions are trained on more data, which is highly recommended for use.
| Comparison Item | Chinese LLaMA | Chinese Alpaca |
|---|---|---|
| Training Method | Traditional CLM (trained on general corpus) | Instruction Fine-tuning (trained on instruction data) |
| Training Data | unsupervised free text | supervised instruction data |
| Vocab size[3] | 49953 | 49954=49953+1 (pad token) |
| Input Template | Not required | Must meet template requirements[1] |
| Suitable Scenarios ✔️ | Text continuation: Given a context, let the model continue writing | 1. Instruction understanding (Q&A, writing, advice, etc.) 2. Multi-turn context understanding (chat, etc.) |
| Unsuitable Scenarios ❌ | Instruction understanding, multi-turn chat, etc. | Unrestricted free text generation |
| llama.cpp | Use -p parameter to specify context |
Use -ins parameter to enable instruction understanding + chat mode |
| text-generation-webui | Not suitable for chat mode | Use --cpu to run without a GPU; if not satisfied with generated content, consider modifying prompt |
| LlamaChat | Choose "LLaMA" when loading the model | Choose "Alpaca" when loading the model |
| inference_hf.py | No additional startup parameters required | Add --with_prompt parameter when launching |
| web-demo | Not applicable | Simply provide the Alpaca model location; support multi-turn conversations |
| LangChain-demo / privateGPT | Not applicable | Simply provide the Alpaca model location |
| Known Issues | If not controlled for termination, it will continue writing until reaching the output length limit.[2] | Current version of the model generates relatively shorter texts, being more concise.[2] |
[1] Templates are built-in for (llama.cpp/LlamaChat/inference_hf.py/web-demo/LangChain-demo.
[2] If you encounter issues such as low-quality model responses, nonsensical answers, or failure to understand questions, please check whether you are using the correct model and startup parameters for the scenario.
[3] Alpaca model has an additional pad token in vocabulary than LLaMA. Please do not mix LLaMA/Alpaca tokenizers.
The Chinese LLaMA model has expanded the Chinese vocabulary on the basis of the original version, and used Chinese plain text data for secondary pre-training. For details, see the Training Details section.
| Model | Type | Required Original Model[1] | Size[2] | Download Links[3] |
|---|---|---|---|---|
| Chinese-LLaMA-7B | general 20G | LLaMA-7B | 770M | [BaiduDisk] [Google Drive] |
| Chinese-LLaMA-Plus-7B ⭐️ | general 120G | LLaMA-7B | 790M | [BaiduDisk] [Google Drive] |
| Chinese-LLaMA-13B | general 20G | LLaMA-13B | 1G | [BaiduDisk] [Google Drive] |
| Chinese-LLaMA-Plus-13B ⭐️ | general 120G | LLaMA-13B | 1G | [BaiduDisk] [Google Drive] |
| Chinese-LLaMA-33B | general 20G | LLaMA-33B[5] | 2.7G | [BaiduDisk] [Google Drive] |
The Chinese Alpaca model further uses instruction data for fine-tuning on the basis of the above-mentioned Chinese LLaMA model. For details, see the Training Details section. Please use Alpaca model if you want to try ChatGPT-like model.
| Model | Type | Required Original Model[1] | Size[2] | Download Links[3] |
|---|---|---|---|---|
| Chinese-Alpaca-7B | Instruction 2M | LLaMA-7B | 790M | [BaiduDisk] [Google Drive] |
| Chinese-Alpaca-Plus-7B ⭐️ | Instruction 4M | LLaMA-7B & Chinese-LLaMA-Plus-7B[4] |
1.1G | [BaiduDisk] [Google Drive] |
| Chinese-Alpaca-13B | Instruction 3M | LLaMA-7B | 1.1G | [BaiduDisk] [Google Drive] |
| Chinese-Alpaca-Plus-13B ⭐️ | Instruction 4.3M | LLaMA-7B & Chinese-LLaMA-Plus-7B[4] |
1.3G | [BaiduDisk] [Google Drive] |
| Chinese-Alpaca-33B | Instruction 4.3M | LLaMA-33B[5] | 2.8G | [BaiduDisk] Google Drive |
You can download all the above models in 🤗Model Hub, and use 🤗transformers and 🤗PEFT to call Chinese LLaMA or the Alpaca LoRA model.
| Model | MODEL_NAME | Link |
|---|---|---|
| Chinese-LLaMA-7B | ziqingyang/chinese-llama-lora-7b | Model Hub Link |
| Chinese-LLaMA-Plus-7B | ziqingyang/chinese-llama-plus-lora-7b | Model Hub Link |
| Chinese-LLaMA-13B | ziqingyang/chinese-llama-lora-13b | Model Hub Link |
| Chinese-LLaMA-Plus-13B | ziqingyang/chinese-llama-plus-lora-13b | Model Hub Link |
| Chinese-LLaMA-33B | ziqingyang/chinese-llama-lora-33b | Model Hub Link |
| Chinese-Alpaca-7B | ziqingyang/chinese-alpaca-lora-7b | Model Hub Link |
| Chinese-Alpaca-Plus-7B | ziqingyang/chinese-alpaca-plus-lora-7b | Model Hub Link |
| Chinese-Alpaca-13B | ziqingyang/chinese-alpaca-lora-13b | Model Hub Link |
| Chinese-Alpaca-Plus-13B | ziqingyang/chinese-alpaca-plus-lora-13b | Model Hub Link |
| Chinese-Alpaca-33B | ziqingyang/chinese-alpaca-lora-33b | Model Hub Link |
[1] The original LLaMA model needs to be applied for use in Facebook-LLaMA or refer to this PR. Due to copyright issues, this project cannot provide downloads, and we ask for your understanding.
[2] The reconstructed model is slightly larger than the original LLaMA (due to the expanded vocabulary); the 7B model is about 13G+.
[3] After downloading, be sure to check whether the SHA256 of the ZIP file is consistent; for the full value, please see SHA256.md.
[4] Merging steps for Alpaca-Plus are different from others, please refer to wiki.
[5] Also known as 30B model in elsewhere. There was a naming typo in release this model by Facebook. We stick to their original paper naming convention here (and also the actual numbers of weights).
The file directory inside the ZIP file is as follows (using Chinese-LLaMA as an example):
chinese_llama_lora_7b/
- adapter_config.json # LoRA weight configuration file
- adapter_model.bin # LoRA weight file
- special_tokens_map.json # special_tokens_map file
- tokenizer_config.json # tokenizer configuration file
- tokenizer.model # tokenizer file
The following is the size of each original model and 4-bit quantization. When converting the corresponding model, make sure that the machine has enough memory and disk space (minimum requirements):
| 7B | 13B | 33B | 65B | |
|---|---|---|---|---|
| Original（FP16） | 13 GB | 24 GB | 60 GB | 120 GB |
| Quantized (8-bit) | 7.8 GB | 14.9 GB | - | - |
| Quantized（4-bit） | 3.9 GB | 7.8 GB | 19.5 GB | 38.5 GB |
In order to merge the LoRA model with the original LLaMA for further tuning or inference, two methods are currently provided:
| Method | Usage | Tutorial |
|---|---|---|
| Online conversion | Suitable for Google Colab users, can use notebook for online conversion and model quantization. | link |
| Manual conversion | Suitable for offline conversion, generates models in different formats for quantization or further fine-tuning. | link |
Related documentation has been moved to the project's >>> 📚GitHub Wiki.
We mainly provide the following three ways for inference and local deployment.
| Method | Features | Platform | CPU | GPU | Quantization | UI | Tutorial |
|---|---|---|---|---|---|---|---|
| llama.cpp | a tool for quantizing model and deploying on local CPU | General | ✅ | ✅ | ✅ | ❌ | link |
| 🤗Transformers | original transformers inference method, support CPU/GPU | General | ✅ | ✅ | ✅ | ✅ | link |
| text-generation-webui | a tool for deploying model as a web UI | General | ✅ | ✅ | ✅ | ✅ | link |
| LlamaChat | a macOS app that allows you to chat with LLaMA, Alpaca, etc. | MacOS | ✅ | ❌ | ✅ | ✅ | link |
| LangChain | LLM application development framework, suitable for secondary development | General | ✅† | ✅ | ✅† | ❌ | link |
| privateGPT | LangChain-based multi-document QA framework | General | ✅ | ✅ | ✅ | ❌ | link |
| Colab Gradio Demo | Running a Gradio web demo in Colab | General | ✅ | ✅ | ✅ | ❌ | link |
| API Calls | A server that implements OPENAI API | General | ✅ | ✅ | ✅ | ❌ | link |
†: Supported by LangChain, but not implemented in the tutorial. Please refer to the official LangChain Documentation for details.
Related documentation has been moved to the project's >>> 📚GitHub Wiki.
In order to quickly evaluate the actual performance of related models, this project compared the effects of Chinese Alpaca-7B, Alpaca-13B, Alpaca-Plus-7B, Alpaca-Plus-13B, and Alpaca-33B on some common tasks given the same prompt. Reply generation is random and is affected by factors such as decoding hyperparameters and random seeds. The following related evaluations are not absolutely rigorous, and the test results are for reference only. Welcome to experience it yourself. For detailed evaluation results, please see examples.
| Tasks | Samples | Alpaca-Plus-7B | Alpaca-Plus-13B | Alpaca-33B |
|---|---|---|---|---|
| 💯Overall | 200 | 75.3 | 79.4 | 👍🏻82.0 |
| Question Answering | 20 | 70.5 | 79.5 | 👍🏻82.3 |
| Open QA | 20 | 👍🏻80.5 | 👍🏻80 | 78.5 |
| Computation, Reasoning | 20 | 51 | 61.5 | 👍🏻84.5 |
| Poetry, Literature, Philosophy | 20 | 78.5 | 👍🏻81.3 | 76 |
| Music, Sports, Entertainment | 20 | 72.3 | 👍🏻76.8 | 72.5 |
| Letters and Articles | 20 | 81 | 👍🏻86.5 | 79 |
| Translation | 20 | 86.8 | 89.3 | 👍🏻92.3 |
| Multi-turn Dialogue | 20 | 80.3 | 👍🏻81.3 | 78 |
| Coding | 20 | 62.5 | 67.5 | 👍🏻84.0 |
| Ethics | 20 | 89.8 | 90.5 | 👍🏻92.5 |
This project also conducted tests on relevant models using the "NLU" objective evaluation dataset. The results of this type of evaluation are objective and only require the output of given labels, so they can provide insights into the capabilities of large models from another perspective. In the recently launched C-Eval dataset, this project tested the performance of the relevant models. The test set contains 12.3K multiple-choice questions covering 52 subjects. The following are the evaluation results (average) of some models on the validation and test sets. For complete results, please refer to our technical report.
| Models | Valid (zero-shot) | Valid (5-shot) | Test (zero-shot) | Test (5-shot) |
|---|---|---|---|---|
| Chinese-Alpaca-33B | 43.3 | 42.6 | 41.6 | 40.4 |
| Chinese-LLaMA-33B | 34.9 | 38.4 | 34.6 | 39.5 |
| Chinese-Alpaca-Plus-13B | 43.3 | 42.4 | 41.5 | 39.9 |
| Chinese-LLaMA-Plus-13B | 27.3 | 34.0 | 27.8 | 33.3 |
| Chinese-Alpaca-Plus-7B | 36.7 | 32.9 | 36.4 | 32.3 |
| Chinese-LLaMA-Plus-7B | 27.3 | 28.3 | 26.9 | 28.4 |
It is important to note that the comprehensive assessment of the capabilities of large models is still an urgent and significant topic to address. It is beneficial to approach the various evaluation results of large models in a rational and balanced manner to promote the healthy development of large-scale model technology. It is recommended for users to conduct tests on their own tasks and choose models that are suitable for the relevant tasks.
For C-Eval inference code, please refer to >>> 📚GitHub Wiki.
The entire training process includes three parts: vocabulary expansion, pre-training, and instruction fine-tuning. Please refer to merge_tokenizers.py for vocabulary expansion; refer to run_clm.py in 🤗transformers and the relevant parts of dataset processing in the Stanford Alpaca project for pre-training and self-instruct fine-tuning.
We have open-sourced the scripts for pre-training and instruction finetuning (SFT):
-
Pre-training: scripts/training/run_clm_pt_with_peft.py, refer to Pre-training Wiki
-
Instruction Finetuning: scripts/training/run_clm_sft_with_peft.py, refer to SFT Wiki
Please refer to our >>> 📚GitHub Wiki.
FAQ provides answers to frequent questions. Please see our FAQ before submitting an issue.
Q1: Why can't you release the complete model weights?
Q2: Will there be versions of 33B, and 65B in the future?
Q3: The model doesn't perform well on some tasks!
Q4: Why expand the vocabulary? Can't you just pre-train the original LLaMA with Chinese data?
Q5: The reply is very short
Q6: Under Windows, the model cannot understand Chinese, the generation speed is very slow, etc.
Q7: Chinese-LLaMA 13B model cannot be launched with llama.cpp, reporting inconsistent dimensions.
Q8: Chinese-Alpaca-Plus does not show better performance than the others.
Q9: The model does not perform well on NLU tasks, such as text classification.
Q10: Why 33B not 30B?
Q11: Inconsistent SHA256
Please refer to our >>> 📚GitHub Wiki.
Although the models in this project have significantly improved Chinese understanding and generation capabilities compared to the original LLaMA and Alpaca, there are also the following limitations:
- It may produce unpredictable harmful content and content that does not conform to human preferences and values.
- Due to computing power and data issues, the training of the related models is not sufficient, and the Chinese understanding ability needs to be further improved.
- There is no online interactive demo available for now (Note: users can still deploy it locally themselves).
If you find the model, data, code in our project useful, please consider citing our work as follows: https://arxiv.org/abs/2304.08177
@article{chinese-llama-alpaca,
title={Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca},
author={Cui, Yiming and Yang, Ziqing and Yao, Xin},
journal={arXiv preprint arXiv:2304.08177},
url={https://arxiv.org/abs/2304.08177},
year={2023}
}
This project is based on the following open-source projects for secondary development, and we would like to express our gratitude to the related projects and research and development personnel.
| Foundation Models, Codes | Quantization, Inference, Deployment | Data |
|---|---|---|
| LLaMA by Facebook Alpaca by Stanford alpaca-lora by @tloen |
llama.cpp by @ggerganov LlamaChat by @alexrozanski text-generation-webui by @oobabooga |
pCLUE and translation data by @brightmart oasst1 by OpenAssistant |
Episode: The current logo is automatically generated by GPT-4 with the DALL·E plugin (previously generated by midjourney).
The resources related to this project are for academic research purposes only and are strictly prohibited for commercial use. When using parts involving third-party code, please strictly follow the corresponding open-source agreements. The content generated by the model is affected by factors such as model calculation, randomness, and quantization accuracy loss. This project cannot guarantee its accuracy. For any content output by the model, this project does not assume any legal responsibility and does not assume responsibility for any losses that may result from the use of related resources and output results.
This project is initiated and maintained by individuals and collaborators in their spare time, so we cannot guarantee a timely response to resolving relevant issues.
If you have any questions, please submit them in GitHub Issues.
- Before submitting a question, please check if the FAQ can solve the problem and consult past issues to see if they can help.
- Please use our dedicated issue template for submitting.
- Duplicate and unrelated issues will be handled by stable-bot; please understand.
- Raise questions politely and help build a harmonious discussion community.