Performance slowdown when paired with ControlNet

[JorgeR81]

Lately, I noticed generation is slower when I use InstantID + ControlNet

It never was that fast to begin with, but it seems it got slower ( at the KSampler stage ).
GPU activity is higher, closer to 100%, when I use them together.
And I think it has a tendency to get slower after generating multiple times.

I haven't used that workflow in a while, and since then I upgraded from pytorch 2.1.0 to pytorch 2.3.1
And Comfy UI also made some changes to improve performance in Flux, which slowed some workflows, for some users.
Not sure if it's got something to do with that ?

---

GTX 1070
32GB RAM
Windows 10
CUDA 12.1

[JorgeR81]

This only happens when 2 ControlNets are used ( InstantID + Depth + Canny )

So I can get similar results, using only InstantID + Depth ( with increased strength ).
I can also use FaceID + Depth + Canny, with no issues.
I can even use FaceID + InstantID + Depth, with with no issues.

The problem is: InstantID + Depth + Canny

The speed difference is really noticeable, comparing these workflows.
There is definitely an issue here.