Debiasing Scores and Prompts of 2D Diffusion for View-consistent Text-to-3D Generation (D-SDS) | NeurIPS 2023

🎭 Why D-SDS?

Debiased Score Distillation Sampling (D-SDS) offers a solution when SDS methods, such as DreamFusion, Magic3D, SJC, etc., don't produce the 3D results you're aiming for. If you've faced issues with artifacts or multiple faces, D-SDS is designed to overcome these challenges through two key mechanisms: Score Debiasing and Prompt Debiasing. For a comprehensive understanding of these processes, we recommend delving into our paper.

Below are the results with D-SDS on ThreeStudio implementation of DreamFusion:

Prompt	SDS (DreamFusion)	Debiased-SDS (Ours)
a colorful toucan with a large beak
a kangaroo wearing boxing gloves

Below are the results with D-SDS on SJC:

SDS (SJC)	Debiased-SDS (Ours)

🐧 How to Use D-SDS

An amazing project, ThreeStudio, has already integrated our method in its main branch. Note that the hyperparameter varies depending on the guidance used. For example, [0, 2.0, 8.0, 10000] is suitable for score debiasing in Stable Diffusion guidance (as described in our paper), while [0, 0.5, 2.0, 10000] is suitable for DeepFloyd-IF guidance. To activate our method, here are some examples:

# Sampling with score debiasing
python launch.py --config configs/dreamfusion-if.yaml --train --gpu 0 system.prompt_processor.prompt="a colorful toucan with a large beak" system.guidance.grad_clip=[0,0.5,2.0,10000]

# Sampling with prompt debiasing
python launch.py --config configs/dreamfusion-if.yaml --train --gpu 0 system.prompt_processor.prompt="a colorful toucan with a large beak" system.prompt_processor.use_prompt_debiasing=true prompt_debiasing_mask_ids=[2]

# Sampling with score & prompt debiasing
python launch.py --config configs/dreamfusion-if.yaml --train --gpu 0 system.prompt_processor.prompt="a colorful toucan with a large beak" system.guidance.grad_clip=[0,0.5,2.0,10000] system.prompt_processor.use_prompt_debiasing=true system.prompt_processor.prompt_debiasing_mask_ids=[2]

You can also find the manual at Tips on Improving Quality.

Acknowledgements

We appreciate the contributions of the great public projects, SJC and ThreeStudio.

Cite as

@article{hong2023debiasing,
  title={Debiasing scores and prompts of 2d diffusion for robust text-to-3d generation},
  author={Hong, Susung and Ahn, Donghoon and Kim, Seungryong},
  journal={arXiv preprint arXiv:2303.15413},
  year={2023}
}