High-quality zero-shot lipsync pipeline built on MuseTalk

[mvoodarla]

Hey folks! My team has been exploring zero-shot lipsyncing for a bit and we think we've improved on MuseTalk's quality quite a bit by using LivePortrait to neutralize expression and CodeFormer to enhance. Here's an example.

short.mp4

We wrote a technical blog on it: https://www.sievedata.com/blog/sievesync-zero-shot-lipsync-api-developers

Hope to put out an OSS repo soon too :)

Anything we don't talk about in the blog that we should in our repo release?

[eoffermann]

I like this - using CodeFormer to do cleanup as a late step in the process is great. I'm doing something right now that leverages MuseTalk as part of a content automation platform and having used all of the packages you mention, from MediaPipe through to CodeFormer, this feels like a really intuitive solution. Thanks for sharing!

[mvoodarla]

Of course!

[mvoodarla]

we just put out a repo here! https://github.com/sieve-community/sievesync

[iv2985]

To anyone curious, they are trying to sell a service API for $0.50/min...

[min-star]

but there still have teeth problem!the teeth is vague!