You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
pytorch default dataloader has each worker load one batch, so with a large batch size like batch_size=64, every worker is loading 64 windows sequentially, and we have to wait for the first worker to finish an entire batch before the training can start.
Maybe it'd be better to form a batch by combining items across workers, given that loading from GCS can be slow?
Or maybe we should switch to Weka and find a good way to sync between GCS and Weka, that could solve the problem.
The text was updated successfully, but these errors were encountered:
pytorch default dataloader has each worker load one batch, so with a large batch size like batch_size=64, every worker is loading 64 windows sequentially, and we have to wait for the first worker to finish an entire batch before the training can start.
Maybe it'd be better to form a batch by combining items across workers, given that loading from GCS can be slow?
Or maybe we should switch to Weka and find a good way to sync between GCS and Weka, that could solve the problem.
The text was updated successfully, but these errors were encountered: