You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
wildebeest would be more useful for a wider range of problems if it could use multiprocessing as well as multithreading.
Describe the solution you'd like
First split into multiple processes, perhaps the number of cores available by default, and then into multiple threads within each core, a la https://github.com/rom1504/img2dataset. I believe this change would never substantially hurt speed and would substantially help in jobs that are somewhat CPU-constrained, e.g. when we are resizing images as we download them.
Describe alternatives you've considered
We could use async, but that would make it harder for users to plug in their own operations.
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
wildebeest
would be more useful for a wider range of problems if it could use multiprocessing as well as multithreading.Describe the solution you'd like
First split into multiple processes, perhaps the number of cores available by default, and then into multiple threads within each core, a la https://github.com/rom1504/img2dataset. I believe this change would never substantially hurt speed and would substantially help in jobs that are somewhat CPU-constrained, e.g. when we are resizing images as we download them.
Describe alternatives you've considered
We could use async, but that would make it harder for users to plug in their own operations.
The text was updated successfully, but these errors were encountered: