[Errno 24] Too many open files error

[binays]

I am training a T5 model with my custom data. Training goes fine. While the model is making predictions, during the "Decoding Outputs" step, it throws an error OSError: [Errno 24] Too many open files: '/tmp/pymp-xzr384iw'

Generating outputs: 100%|█████████████████████████| 1/1 [00:00<00:00, 1.47it/s]
Decoding outputs: 0%| | 0/2 [00:01<?, ?it/s]
Traceback (most recent call last):
File "model_1_test.py", line 33, in
File "/home/dahalb1/sentence_modeling/venv/lib/python3.6/site-packages/simpletransformers/t5/t5_model.py", line 846, in predict
File "/home/dahalb1/sentence_modeling/venv/lib/python3.6/site-packages/tqdm/std.py", line 1166, in iter
File "/usr/lib/python3.6/multiprocessing/pool.py", line 320, in
File "/usr/lib/python3.6/multiprocessing/pool.py", line 735, in next
File "/usr/lib/python3.6/multiprocessing/pool.py", line 424, in _handle_tasks
File "/usr/lib/python3.6/multiprocessing/connection.py", line 206, in send
File "/usr/lib/python3.6/multiprocessing/reduction.py", line 51, in dumps
File "/home/dahalb1/sentence_modeling/venv/lib/python3.6/site-packages/torch/multiprocessing/reductions.py", line 322, in reduce_storage
File "/usr/lib/python3.6/multiprocessing/reduction.py", line 191, in DupFd
File "/usr/lib/python3.6/multiprocessing/resource_sharer.py", line 48, in init
OSError: [Errno 24] Too many open files
Traceback (most recent call last):
File "/usr/lib/python3.6/weakref.py", line 624, in _exitfunc
File "/usr/lib/python3.6/weakref.py", line 548, in call
File "/usr/lib/python3.6/tempfile.py", line 938, in _cleanup
File "/usr/lib/python3.6/shutil.py", line 482, in rmtree
File "/usr/lib/python3.6/shutil.py", line 480, in rmtree
OSError: [Errno 24] Too many open files: '/tmp/tmprwm6yj5lwandb'
Traceback (most recent call last):
File "/usr/lib/python3.6/weakref.py", line 624, in _exitfunc
File "/usr/lib/python3.6/weakref.py", line 548, in call
File "/usr/lib/python3.6/tempfile.py", line 938, in _cleanup
File "/usr/lib/python3.6/shutil.py", line 482, in rmtree
File "/usr/lib/python3.6/shutil.py", line 480, in rmtree
OSError: [Errno 24] Too many open files: '/tmp/tmptt7g7ivwwandb-media'
Traceback (most recent call last):
File "/usr/lib/python3.6/multiprocessing/util.py", line 262, in _run_finalizers
File "/usr/lib/python3.6/multiprocessing/util.py", line 186, in call
File "/usr/lib/python3.6/shutil.py", line 482, in rmtree
File "/usr/lib/python3.6/shutil.py", line 480, in rmtree
OSError: [Errno 24] Too many open files: '/tmp/pymp-xzr384iw'

I'm training on a RTX-8000 GPU with Ubuntu installed.

[ThilinaRajapakse]

This should be resolved now. Multiprocessing is now turned off by default due to stability concerns.

[ahmad-abdellatif]

Is there any update on this issue? I am facing the same problem while training the T5 model using SimpleTransformers v0.61.4.

[WeiShi-9]

I solved this problem by running "ulimit -n 400000" before my code