3D character generation DreamGaussian live rigging saved scenes etc.

[liamzebedee]

This contains the code to automatically create 3D characters while we are running the show. The character heads are generated by a custom AI pipeline written in Python, using SDXL and Dreamgaussian, live imported into the show at runtime and rigged to the "default guy" model.

This works through changing the show:

• 3d scenes are generated in the background if an AI character is detected in the script
• when a 3d character is detected, we run a separate "character voting" scene
• we then play the scene

Lots of other cool features:

• save and replay scenes from the filesystem - including dialogue, voice tracks, ai generated art (dimensions, characters)
• entire history of the show is saved as we run the main loop
• PlayScene now perfectly starts from the same place every time - resetting rick/morty to their positions, resetting camera angles, resetting the AI art/dimension controllers (rewritten)

images.

3d characters

A 3d donald trump generated at runtime:

Sam altman telling codebullet about AGI

Gordon Ramsey and Dutch guys:

The new character voting chamber:

Live reaction bar:

The saved files and scenes written to disk:

new API's

// Loading scenes
var scene = RickAndMortyScene.ReadFromDir("scene-obama-ai");
await RunScene(scene);

// Saving scenes
var scene = await CreateScene(firstPrompt, "me", "banana", "me", usingVoiceActing);
await scene.WriteToDir();

// Run topic vote
TopicVoteResults voteResults = await RunTopicVote(nextSceneTask, 30f);
var scene = CreateScene(
    voteResults.topic,
    voteResults.author,
    voteResults.backupTopic,
    voteResults.backupAuthor,
    usingVoiceActing
);

// Run new character vote.
CharacterVoteResults voteRes = await RunCharacterVote(
    "barack obama",
    AIHeadRigger.GetGenerationsForCharacter("barack obama"),
    5000
);

// Hiding/showing different UI's
SetUI(topicVotingUI);
SetUI(charVotingUI);
SetUI(episodeStuffUI); // titles, subtitles, etc
SetUI(null); // show none

the 3d generation server (python)

• generates 4x candidate images in parallel using SDXL, with prompt/antiprompt/reference image configured
• feeds these 4x images to a dreamgaussian model, generating 4x 3d models in parallel
• outputs .zip containing 3d model (.obj) and textures (.mtl, .png)
• generation is run in the Replicate cloud
• benchmarking:
  • using Replicate A40
  • model time - 120s (cold boot) + 30s (first run) + 120s (gen/inference) = 270s
  • first run goes for an extra 30s due to the model's cog container downloading some extra files
• everything is based on the raw filesystem - no need to make it complex

Filesystem layout:

The boys:

Reference of 2d SDXL to 3d DreamGaussian quality:

CLI view: