Issue with Realistic Models

[paulo-coronado]

Hello! I am trying to apply Cutoff to realistic models, such as RealisticVision2.0 and the tokens doesn't work as expected. Any ideas? I think it is because the model was not trained with this colored regions, so it refuses to generate as requested.

This is an image generated with the following prompt:

beautiful european model, white shirt with green tie, black shoes, blue hair, yellow eyes, pink skirt, full body shot, walking on a street

[BlenderNeko]

Unfortunately your mileage may vary depending on the model that you're using. After playing around with realistic vision for a bit, it seems that realistic vision appears particularly bad for this, at least when it comes to colors.

[arcusmaximus]

For A1111 there's the Regional Prompter extension whose region specification by prompt feature can be used for the same goal (preventing color bleeding) but with a completely different approach. Given a prompt like

beautiful european model, tie BREAK
green, tie

it can find the area in the image that corresponds to the tie keyword and apply the green prompt just to that. It also previews the masks that it found, and lets you grow/shrink them by tweaking their threshold values.

Here's a quick one I did with, yes, RealisticVision 2.0:

So this approach seems more robust than patching embedding vectors - case in point, A1111 Cutoff gave me rather mixed results even for anime. Having looked at ComfyUI for the first time today, though, I'm not sure something similar exists there yet.