Multi GPU Training

[thetznecker]

Is there a real time benefit when training with multiple GPUS or is it just about adding up VRAM for bigger batch sizes?

Or would it be better to train 1 model on device 0 and 1 model on device 1?

Also, has someone got the commands to start the multi-gpu training correctly?
We're getting tensor.detach() errors and can't find any related information about it.