ClipVision Enhancer

[cubiq]

I've added a ClipVision enhancer node, it's very experimental and I'm not even sure the math behind it is 100% correct but the preliminary results are pretty good. In the image below you can see in the middle the enhanced version, on the left is standard IPAdapter (on the right the reference image).

I am basically tiling the image, generate the embeds for each tile and then I recompose embeds in same position they were in the original image and finally pool everything to the default embed size. I hope this is explained by the image below.

@xiaohu2015 do you think this could be used in training too? Have you tried something like this?

[xiaohu2015]

maybe we can use a clip model with higher resolution (e.g. 512x512)

[cubiq]

that would solve all issues :)

Kolors models use 336 and the result seems pretty good, I've seen 448 clip, not sure I remember seeing 512 clip encoders

[krahnikblis]

seems a lot like how Disco Diffusion works, with all the cuts of the image pulled apart, warped and augmented, run thru CLIP, then the final embeds are a normed result of all the positional CLIP values collected from all the cuts. you might wanna try wholesale stealing the code from this project (which is a wrapped-up version of disco for Comfy) - the make_cutouts.py script does all the equivalent of what you call tiles here, and the nodes.py file has implementation and control for cutn, innercut, overview, etc... a neat thing about Disco is you can use a bunch of different CLIP models together too, enhancing the semantic vision capability...