EOFError: Ran out of input

[henrypearce4D]

(4d_gaussian_splatting) c:\4d-gaussian-splatting>python train.py --config configs/dynerf/coffee_martini.yaml
Using C:\Users\admin\AppData\Local\torch_extensions\torch_extensions\Cache\py37_cu116 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file C:\Users\admin\AppData\Local\torch_extensions\torch_extensions\Cache\py37_cu116\diff_gaussian_rasterization\build.ninja...
Building extension module diff_gaussian_rasterization...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module diff_gaussian_rasterization...
Optimizing output/N3V/coffee_martini
Output folder: output/N3V/coffee_martini [15/02 19:40:42]
Tensorboard not available: not logging progress [15/02 19:40:42]
Found transforms_train.json file, assuming Blender data set! [15/02 19:40:42]
Reading Training Transforms [15/02 19:40:42]
100%|████████████████████████████████████████████████████████████████████████████| 5100/5100 [00:01<00:00, 4184.53it/s]
Reading Test Transforms [15/02 19:40:44]
100%|██████████████████████████████████████████████████████████████████████████████| 300/300 [00:00<00:00, 2133.85it/s]
Loading Training Cameras [15/02 19:40:45]
Loading Test Cameras [15/02 19:40:46]
Number of points at initialisation :  300000 [15/02 19:40:48]
Training progress:   0%|                                                                     | 0/30000 [00:00<?, ?it/s]Traceback (most recent call last):
  File "train.py", line 403, in <module>
    args.gaussian_dim, args.time_duration, args.num_pts, args.num_pts_ratio, args.rot_4d, args.force_sh_3d, args.batch_size)
  File "train.py", line 83, in training
    for batch_data in training_dataloader:
  File "C:\Users\admin\anaconda3\envs\4d_gaussian_splatting\lib\site-packages\torch\utils\data\dataloader.py", line 444, in __iter__
    return self._get_iterator()
  File "C:\Users\admin\anaconda3\envs\4d_gaussian_splatting\lib\site-packages\torch\utils\data\dataloader.py", line 390, in _get_iterator
    return _MultiProcessingDataLoaderIter(self)
  File "C:\Users\admin\anaconda3\envs\4d_gaussian_splatting\lib\site-packages\torch\utils\data\dataloader.py", line 1077, in __init__
    w.start()
  File "C:\Users\admin\anaconda3\envs\4d_gaussian_splatting\lib\multiprocessing\process.py", line 112, in start
    self._popen = self._Popen(self)
  File "C:\Users\admin\anaconda3\envs\4d_gaussian_splatting\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Users\admin\anaconda3\envs\4d_gaussian_splatting\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\Users\admin\anaconda3\envs\4d_gaussian_splatting\lib\multiprocessing\popen_spawn_win32.py", line 89, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Users\admin\anaconda3\envs\4d_gaussian_splatting\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'training.<locals>.<lambda>'
Training progress:   0%|                                                                     | 0/30000 [00:02<?, ?it/s]

(4d_gaussian_splatting) c:\4d-gaussian-splatting>Using C:\Users\admin\AppData\Local\torch_extensions\torch_extensions\Cache\py37_cu116 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file C:\Users\admin\AppData\Local\torch_extensions\torch_extensions\Cache\py37_cu116\diff_gaussian_rasterization\build.ninja...
Building extension module diff_gaussian_rasterization...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module diff_gaussian_rasterization...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\admin\anaconda3\envs\4d_gaussian_splatting\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\admin\anaconda3\envs\4d_gaussian_splatting\lib\multiprocessing\spawn.py", line 115, in _main
    self = reduction.pickle.load(from_parent)
EOFError: Ran out of input

[Alexander0Yang]

Hi, @henrypearce4D,
The cause of this error is that multiprocessing is not well supported on Windows. You can try to use Linux, or set num_workers=0 (may be very slow).

[henrypearce4D]

@Alexander0Yang Thanks for the suggestions and sorry for the delay in reply.

I tested on windows with set num_workers=0 and was able to train, it did take a very long time though as you said.
I am now testing on linux and the training has started very slow but I need to let it run longer.

I now have a few questions;

1. I have my trained chkpnt30000.pth and was able to render the results using this suggestion How to Visualize or test the training results？ #12 (comment)
2. Is there a command to convert the chkpnt.pth to .ply ?
3. The file size for the coffee_martini dataset for chkpnt30000.pth is 10.1GB, is this as expected? chkpnt7000.pth is also 5.88GB

[EarltShirt]

hey @Alexander0Yang , how did you get it to work ? setting the num_workers=0 didn't change anything, still getting the same error. Couldn't it be due to the fact that one is trying to pickle a lambda function which isn't supported to my knowledge.

AttributeError: Can't pickle local object 'training.<locals>.<lambda>'
Training progress:   0%|                                                                                                          | 0/30000 [00:02<?, ?it/s]

(4d_gaussian_splatting) C:\Users\AD278604\Documents\repos\4d-gaussian-splatting>Using C:\Users\admin-local\AppData\Local\torch_extensions\torch_extensions\Cache\py37_cu116 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file C:\Users\admin-local\AppData\Local\torch_extensions\torch_extensions\Cache\py37_cu116\diff_gaussian_rasterization\build.ninja...
Building extension module diff_gaussian_rasterization...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module diff_gaussian_rasterization...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\AD278604\AppData\Local\anaconda3\envs\4d_gaussian_splatting\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\AD278604\AppData\Local\anaconda3\envs\4d_gaussian_splatting\lib\multiprocessing\spawn.py", line 115, in _main
    self = reduction.pickle.load(from_parent)
EOFError: Ran out of input