Question about the refiner step

[lifeisboringsoprogramming]

I am reading the SDXL paper and found that the refiner is applying to the latent image

but in your codes,

images = pipe_refiner(prompt=prompt, negative_prompt=negative, image=images, num_inference_steps=steps, strength=refiner_strength).images

the input are the images instead of the latents
are they the same?

thanks

[TonyLianLong]

I believe it will re-encode so it's applied on the latents.

The implementation shows that images are transformed to latents prior to processing: https://github.com/huggingface/diffusers/blob/af48bf200860d8b83fe3be92b2d7ae556a3b4111/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl_img2img.py#L841

I believe this is their recommended way to do refinement, as this is in their PR examples.

[lifeisboringsoprogramming]

Thank you for your reply
The PR use

images = pipe(prompt=prompt, output_type="latent").images

before the refiner

there is an output_type="latent" parameter

How much VRAM is needed if we move the pipe to cuda instead of enable_model_cpu_offload?

I do not have enough VRAM so I cannot know

Thanks

[TonyLianLong]

there is an output_type="latent" parameter

In this way you are right, the process of converting the latents to image can be skipped.

How much VRAM is needed if we move the pipe to cuda instead of enable_model_cpu_offload?

With 4 images, you can use 24G GPU Memory.

[lifeisboringsoprogramming]

I have 12G VRAM and cannot even do one image using pipe to cuda, thank you so much.

[TonyLianLong]

Two updates:

1. If the intermediate images are not needed (i.e., we don't want to compare before/after), no decoding and re-encoding between the base generation and refinement stage are used.
2. Offloading can be controlled with environment variables.

[lifeisboringsoprogramming]

I got some results
on the left: using images as refiner input
on the right: using latent as refiner input

the head of the middle guy has some differences.

Thanks

[TonyLianLong]

Thanks for this example. Is using images consistently better than using latents?

[lifeisboringsoprogramming]

I did not do any more testing for that
I think I decided to do the refiner only after knowing how this picture looked is a better workflow