Would multi-threaded solving per branch be an option?

[ultra-sonic]

Hi Timur,

we have brainstormed a bit on how to improve multi-threading in afserver and one question that came to my mind is:
Would multi-threaded solving per branch be an option?

All our jobs usually live inside the leafs of our branches and there are only very few exception. a basic example of how we structured our branches is:
in the root branch there is only a /2d and /3d branch - so jobs inside of those branches should be solveable in parallel by two additional threads and the main thread should only decide which of both branches comes next based on their need right?

in theory one could play that game even for the children of those branches - in our case these are our geo-locations e.g.:

/2d/siteA
/2d/siteB
/2d/siteC
/2d/siteD
/2d/siteE
/3d/siteA
/3d/siteB
/3d/siteC
/3d/siteD
/3d/siteE

modern cpus have 256 cores and having this feature would make afanasy superior to pretty much any other render-manager out there.

what do you think?

cheers
Oli

[ultra-sonic]

@sebastianelsner

[timurhai]

Hi, Oli!
It will be hard to implement multi-thread solving. Solving includes setting task to render, this increments different counts that affects the next solve. Even if we split solve on 2 different stages, find task and set task, find depends on set that increments counts. Counts are everywhere, on render and all its parent pools, jobs and all its parent branches and users.
Different threads can read something at the same time, but if they need to write the same, there should be synchronization mechanisms (semaphores). Lots of semaphores will slow down the system, and the code will be very difficult (not so plain as for a single thread).
One way that I see for now, is to create some pre-find stage and dispatch it on different threads. That pre-find stage should only read data and not write. For example pre-find can find a ready task for each ready job, but it can't find a render for that task, because each task changes renders.