We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Here's the command when trying on 4 machines:
deepspeed --num_gpus 8 --num_nodes 4 --master_addr sphinx5 --hostfile hostfile train.py --config conf/mistral-32gpu.yaml --nnodes 4 --nproc_per_node 8 --training_arguments.fp16 true --training_arguments.per_device_train_batch_size 4 --training_arguments.deepspeed conf/deepspeed/z2-small-conf.json --run_id gpt2-32gpu-demo
Here's the crash error:
sphinx5: File "train.py", line 264, in <module> sphinx5: Traceback (most recent call last): sphinx5: File "train.py", line 264, in <module> sphinx5: Traceback (most recent call last): sphinx5: File "train.py", line 264, in <module> sphinx5: Traceback (most recent call last): sphinx5: Traceback (most recent call last): sphinx5: Traceback (most recent call last): sphinx5: Traceback (most recent call last): sphinx5: File "train.py", line 264, in <module> sphinx5: File "train.py", line 264, in <module> sphinx5: File "train.py", line 264, in <module> sphinx5: File "train.py", line 264, in <module> sphinx5: train() sphinx5: File "train.py", line 122, in train sphinx5: train() sphinx5: File "train.py", line 122, in train sphinx5: train() sphinx5: File "train.py", line 122, in train sphinx5: custom_eval_datasets, lm_dataset = load_datasets(quinfig, paths, tokenizer, overwatch) sphinx5: File "train.py", line 191, in load_datasets sphinx5: train() sphinx5: File "train.py", line 122, in train sphinx5: custom_eval_datasets, lm_dataset = load_datasets(quinfig, paths, tokenizer, overwatch) sphinx5: File "train.py", line 191, in load_datasets sphinx5: train() sphinx5: _preprocess_once_per_machine(quinfig, paths, tokenizer, overwatch) sphinx5: File "train.py", line 256, in _preprocess_once_per_machine sphinx5: File "train.py", line 122, in train sphinx5: train()custom_eval_datasets, lm_dataset = load_datasets(quinfig, paths, tokenizer, overwatch) sphinx5: sphinx5: File "train.py", line 191, in load_datasets sphinx5: custom_eval_datasets, lm_dataset = load_datasets(quinfig, paths, tokenizer, overwatch)raise RuntimeError(f"Forked process exited with status {status[0]}") File "train.py", line 122, in train sphinx5: sphinx5: sphinx5: File "train.py", line 191, in load_datasets sphinx5: RuntimeError_preprocess_once_per_machine(quinfig, paths, tokenizer, overwatch) : Traceback (most recent call last): sphinx5: sphinx5: Forked process exited with status -7train() sphinx5: File "train.py", line 256, in _preprocess_once_per_machine sphinx5: sphinx5: File "train.py", line 264, in <module> sphinx5: File "train.py", line 122, in train sphinx5: custom_eval_datasets, lm_dataset = load_datasets(quinfig, paths, tokenizer, overwatch) sphinx5: File "train.py", line 191, in load_datasets sphinx5: _preprocess_once_per_machine(quinfig, paths, tokenizer, overwatch) sphinx5: File "train.py", line 256, in _preprocess_once_per_machine sphinx5: _preprocess_once_per_machine(quinfig, paths, tokenizer, overwatch) sphinx5: File "train.py", line 256, in _preprocess_once_per_machine sphinx5: custom_eval_datasets, lm_dataset = load_datasets(quinfig, paths, tokenizer, overwatch) sphinx5: File "train.py", line 191, in load_datasets sphinx5: raise RuntimeError(f"Forked process exited with status {status[0]}") sphinx5: custom_eval_datasets, lm_dataset = load_datasets(quinfig, paths, tokenizer, overwatch) sphinx5: File "train.py", line 191, in load_datasets sphinx5: RuntimeError: Forked process exited with status -7 sphinx5: _preprocess_once_per_machine(quinfig, paths, tokenizer, overwatch) sphinx5: File "train.py", line 256, in _preprocess_once_per_machine sphinx5: _preprocess_once_per_machine(quinfig, paths, tokenizer, overwatch)raise RuntimeError(f"Forked process exited with status {status[0]}")
The text was updated successfully, but these errors were encountered:
No branches or pull requests
Here's the command when trying on 4 machines:
Here's the crash error:
The text was updated successfully, but these errors were encountered: