Wav2Lip-HD: Improving Wav2Lip to achieve High-Fidelity Videos

This repository contains code for achieving high-fidelity lip-syncing in videos, using the Wav2Lip algorithm for lip-syncing and the Real-ESRGAN algorithm for super-resolution. The combination of these two algorithms allows for the creation of lip-synced videos that are both highly accurate and visually stunning.

Algorithm

The algorithm for achieving high-fidelity lip-syncing with Wav2Lip and Real-ESRGAN can be summarized as follows:

1. The input video and audio are given to Wav2Lip algorithm.
2. Python script is written to extract frames from the video generated by wav2lip.
3. Frames are provided to Real-ESRGAN algorithm to improve quality.
4. Then, the high-quality frames are converted to video using ffmpeg, along with the original audio.
5. The result is a high-quality lip-syncing video.
6. The specific steps for running this algorithm are described in the Testing Model section of this README.

Testing Model

To test the "Wav2Lip-HD" model, follow these steps:

1. Clone this repository and install requirements using following command (Make sure, Python and CUDA are already installed):

   git clone https://github.com/saifhassan/Wav2Lip-HD.git
   cd Wav2Lip-HD
   pip install -r requirements.txt
   

2. Downloading weights

Model	Directory	Download Link
Wav2Lip	checkpoints/	Link
ESRGAN	experiments/001_ESRGAN_x4_f64b23_custom16k_500k_B16G1_wandb/models/	Link
Face_Detection	face_detection/detection/sfd/	Link
Real-ESRGAN	Real-ESRGAN/gfpgan/weights/	Link
Real-ESRGAN	Real-ESRGAN/weights/	Link

3. Put input video to input_videos directory and input audio to input_audios directory.

4. Open run_final.sh file and modify following parameters:

   filename=kennedy (just video file name without extension)

   input_audio=input_audios/ai.wav (audio filename with extension)

5. Execute run_final.sh using following command:

   bash run_final.sh
   

6. Outputs

• output_videos_wav2lip directory contains video output generated by wav2lip algorithm.
• frames_wav2lip directory contains frames extracted from video (generated by wav2lip algorithm).
• frames_hd directory contains frames after performing super-resolution using Real-ESRGAN algorithm.
• output_videos_hd directory contains final high quality video output generated by Wav2Lip-HD.

Results

The results produced by Wav2Lip-HD are in two forms, one is frames and other is videos. Both are shared below:

Example output frames

Frame by Wav2Lip	Optimized Frame

Example output videos

Video by Wav2Lip	Optimized Video
kennedy_low.mp4	kennedy_hd.mp4
mona_low.mp4	mona_hd-2.mp4

Acknowledgements

We would like to thank the following repositories and libraries for their contributions to our work:

1. The Wav2Lip repository, which is the core model of our algorithm that performs lip-sync.
2. The face-parsing.PyTorch repository, which provides us with a model for face segmentation.
3. The Real-ESRGAN repository, which provides the super resolution component for our algorithm.
4. ffmpeg, which we use for converting frames to video.