hfd

🎈Rust-based interface for Huggingface 🤗 download.

# Download entire public hf repo
./hdf https://huggingface.co/deepseek-ai/DeepSeek-V2

# Download public hf repo with limit num
./hdf https://huggingface.co/microsoft/Florence-2-large -n 10

# Download gated public hf repo with token
./hdf https://huggingface.co/meta-llama/Meta-Llama-3-70B -t hf_xxxxxxxxxx

# Download gated public hf repo with token and save to /data/llm
./hdf https://huggingface.co/meta-llama/Meta-Llama-3-70B -t hf_xxxxxxxxxx -d /data/llm

# Sometimes, use mirror, for example, hf-mirror.com. 
# The following two options are feasible.
./hdf https://huggingface.co/meta-llama/Meta-Llama-3-70B -t hf_xxxxxxxxxx -d /data/llm -m hf-mirror.com
./hdf https://hf-mirror.com/meta-llama/Meta-Llama-3-70B -t hf_xxxxxxxxxx -d /data/llm

For a more convinent user experience, execute:

cat <<EOF | sudo tee -a /etc/security/limits.conf
root soft nofile 20000000
root hard nofile 20000000
*       hard    nofile  20000000
*       soft    nofile  20000000
EOF

cat <<EOF | sudo tee /etc/sysctl.d/bbr.conf
net.core.default_qdisc=fq_codel
net.ipv4.tcp_congestion_control=bbr
net.ipv4.tcp_moderate_rcvbuf = 1
net.ipv4.tcp_mem = '10000000 10000000 10000000'
net.ipv4.tcp_rmem = '1024 4096 16384'
net.ipv4.tcp_wmem = '1024 4096 16384'

net.core.wmem_max = 26214400
net.core.rmem_max = 26214400

fs.file-max = 12000500
fs.nr_open = 20000500
EOF

Tip

v0.2.6 Processing time: 4457.868372796s
v0.2.7 Processing time: 4441.127406732s
v0.2.8 Processing time: 4650.917674426s

• hyperium/hyper#1358
• hyperium/hyper#1358 (comment)
• https://gist.github.com/klausi/f94b9aff7d36a1cb4ebbca746f0a099f
• https://gist.github.com/mustafaturan/47268d8ad6d56cadda357e4c438f51ca