Sets up Rad-Ninjas' LLM On Prem
Add to requirements.yml:
---
- name: llm_on_prem
src: https://github.com/RAD-Ninjas/ansible-role-llm-on-prem.git
scm: git
Install the role
$ ansible-galaxy install -r requirements.yml
Use it:
- role: llm_on_prem
vars:
llm_model_path: lmsys/vicuna-7b-v1.3
llm_model_name: vicuna-7b-v1.3
ai_profile: cpu
# Never store passwords insecurely
ai_huggingface_token: "{{ lookup('community.general.onepassword', 'huggingface-read-token', field='password', vault='YourVaultName') }}"
ai_openai_api_key: "{{ lookup('community.general.onepassword', 'penai-api-key', field='password', vault='YourVaultName') }}"
docker_username: "{{ lookup('community.general.onepassword', 'home-network-docker-username', field='username', vault='HomeAutomation') }}"
docker_password: "{{ lookup('community.general.onepassword', 'home-network-docker-username', field='password', vault='HomeAutomation') }}"
An attempt has been made to only use ansible builtins. However, since it assumes docker is installed and configured (with ample memory and swap), it is best used with geerlingguy/ansible-role-docker
By default, the huggingface token and API keys read as follows:
ai_huggingface_token: "{{ lookup('community.general.onepassword', 'home-network-huggingface-read-token', field='password', vault='YourVaultName') }}"
ai_openai_api_key: "{{ lookup('community.general.onepassword', 'home-network-openai-api-key', field='password', vault='YourVaultName') }}"
Variable | Default | Reason |
---|---|---|
llm_model_path | lmsys/vicuna-7b-v1.3 | Huggingface path to the Model. |
llm_model_name | vicuna-7b-v1.3 | Huggingface name of the model |
ai_profile | cpu | Inference build Profile. cpu, gpu, metal, openvino, etc. |
ai_cpu_threads | 5 | Number of CPU threads to use for CPU inference |
ai_openvino_ir_path | /home/openvino/models/llama-2/ir_model | The path for OpenVino IR |
ai_openvino_device | GPU.1 | The device name for OpenVino |
So far only builtins. The 1password ansible community tasks if you use the 1password lookup on your vars.
Including an example of how to use your role (for instance, with variables passed in as parameters) is always nice for users too:
- hosts: llm_on_prem_hosts
roles:
- role: llm_on_prem
vars:
# The Profile as per RAD-Ninjas scripts. CPU, GPU, metal (Mac), etc.
ai_profile: cpu
ai_huggingface_token: "{{ lookup('community.general.onepassword', 'huggingface-read-token', field='password', vault='MyVault') }}"
ai_openai_api_key: "{{ lookup('community.general.onepassword', 'openai-api-key', field='password', vault='MyVault') }}"
docker_username: "{{ lookup('community.general.onepassword', 'docker', field='username', vault='MyVault') }}"
docker_password: "{{ lookup('community.general.onepassword', 'docker', field='password', vault='MyVault') }}"
This ansible role should work on any machine that is compatible with docker.
If you use a container-in-container option such as Proxmox or other Hypervisors, the container you use must have Nesting (so docker can run in the first place) and NFS support (so docker can mount its own volumes). And it may be best if you either remove AppArmor from the container or configure it to run the specific docker processes on llm-on-prem.
BSD
This role was written David Martinez with help with the amazing AI folks at RAD-Ninjas.