Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

3D character generation DreamGaussian live rigging saved scenes etc. #17

Open
wants to merge 64 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
64 commits
Select commit Hold shift + click to select a range
f84df93
try running on macOS
liamzebedee Oct 16, 2023
0e28b52
macOS env vars not working, trying GlobalConfig singleton
liamzebedee Oct 16, 2023
4d789f8
add GlobalConfig, load config from Assets/config.json if it exists, e…
liamzebedee Oct 16, 2023
1ec41f2
refactor: use singleton config for OPENAI_API_KEY
liamzebedee Oct 16, 2023
5449d06
refactor: use singleton config for FAKE_YOU API login
liamzebedee Oct 16, 2023
ee419f9
change exec order of WholeThingManager, so we load singleton config b…
liamzebedee Oct 16, 2023
8f0feb5
local image generation using sd
liamzebedee Oct 16, 2023
7b684f9
update doc
liamzebedee Oct 16, 2023
7df343a
test XL model on CUDA
liamzebedee Oct 17, 2023
fd08d44
commit highres batman prompts
liamzebedee Oct 17, 2023
5a0ac03
3d generated headshots pipeline
liamzebedee Oct 31, 2023
f83b295
add localimagegen code
liamzebedee Nov 6, 2023
da210f4
add AiHead generator and rigger scripts
liamzebedee Nov 6, 2023
100ce0a
save/load scenes locally
liamzebedee Nov 9, 2023
b7e996b
add the unity recorder package so we can record scenes as mp4's
liamzebedee Nov 9, 2023
d324615
reset AI art controllers on scene
liamzebedee Nov 13, 2023
4428093
main loop refactor working reliably
liamzebedee Nov 13, 2023
4431164
fix: unknown import in fakeyou
liamzebedee Nov 13, 2023
0844dc1
reset cameras to defaults for scenes
liamzebedee Nov 13, 2023
a3ec685
.
liamzebedee Nov 13, 2023
4c2dc69
add sample scene to replay
liamzebedee Nov 13, 2023
54aadc9
configure default to replay sample scene
liamzebedee Nov 13, 2023
109bc39
merge master
liamzebedee Nov 13, 2023
aa95bf3
fix: RandomCameraDance should use Awake so it goes after WholeThingMa…
liamzebedee Nov 13, 2023
6f99941
fix: rick and morty always start in garage
liamzebedee Nov 13, 2023
251d336
.
liamzebedee Nov 13, 2023
0360df9
fix rick and morty position in the editor
liamzebedee Nov 13, 2023
529c4e6
integrate the 3d character generation into live rendered scenes
liamzebedee Nov 21, 2023
a5fb5a6
lit shader - litty lit littttt
liamzebedee Nov 21, 2023
a1b0aab
add a voting chamber
liamzebedee Nov 21, 2023
7685c1a
.
liamzebedee Nov 21, 2023
a5397f1
add who's that pokemon exp
liamzebedee Nov 27, 2023
e6770bd
playscene now resets cameras/main character positions at start of eve…
liamzebedee Nov 27, 2023
32fe62b
youtubechat: count the char:# votes
liamzebedee Nov 27, 2023
ccac88d
character voting chamber + UI refactors
liamzebedee Nov 27, 2023
78b4d63
missing pokemon scenes
liamzebedee Nov 27, 2023
5b5622a
add videorecorder for offline mp4 recording of scenes
liamzebedee Nov 27, 2023
b11bba7
resave main scene - didn't save properly before
liamzebedee Nov 27, 2023
3688b55
working test workflow 2
liamzebedee Nov 27, 2023
408cd38
test “3d character generation to voting flow” w/ “background scene pl…
liamzebedee Nov 28, 2023
8e645da
refactoring
liamzebedee Dec 4, 2023
c0989ea
- [x] Refactor the voice acting into its own function
liamzebedee Dec 4, 2023
c3f0218
- [x] Fix the scenes add camera instructions
liamzebedee Dec 4, 2023
b251d5e
* add UI dropdown to swap run mode (mainloop, replay scene, etc)
liamzebedee Dec 5, 2023
24c8d9c
add remaining dropdown stuff
liamzebedee Dec 5, 2023
600a518
add liam's test channel in comment
liamzebedee Dec 12, 2023
6e425ca
add mock chat script (so ppl don't keep asking me why my stream sucks)
liamzebedee Dec 12, 2023
4239c6a
* doc share drive
liamzebedee Dec 12, 2023
998cb68
* add audience ratings bar aka THE BULLET METER
liamzebedee Dec 12, 2023
d87217d
delete old events code for audience ratings
liamzebedee Dec 12, 2023
b294a00
fix
liamzebedee Dec 12, 2023
fb09fe8
add % label to audience bar
liamzebedee Dec 12, 2023
6a65fc1
add generation server
liamzebedee Dec 12, 2023
3eb7eb7
fix camera shots prompt
liamzebedee Feb 7, 2024
cb4beec
add ui for custom script
liamzebedee Feb 10, 2024
b362b4b
.
liamzebedee Mar 20, 2024
31c6694
move data dir
liamzebedee Mar 20, 2024
f8bce67
.
liamzebedee Mar 20, 2024
d3cefe3
refactor
liamzebedee Mar 24, 2024
54c0546
.
liamzebedee Mar 25, 2024
7e6a58d
.
liamzebedee Mar 25, 2024
11c7535
.
liamzebedee Mar 25, 2024
780ebe3
fix final bugs in audioclip lag while recording, other stuff
liamzebedee Mar 27, 2024
e19c1e7
.
liamzebedee Mar 27, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
GenerationServer/
BigLezForever/

# This .gitignore file should be placed at the root of your Unity project directory
#
# Get latest from https://github.com/github/gitignore/blob/main/Unity.gitignore
Expand Down Expand Up @@ -71,8 +74,18 @@ crashlytics-build.properties
/[Aa]ssets/[Ss]treamingAssets/aa.meta
/[Aa]ssets/[Ss]treamingAssets/aa/*

# Configuration.
/[Aa]ssets/config.json*

/[Aa]ssets/Liam'z Experiments

node_modules/

OutputScripts/
.vsconfig

Research/

data/prod
!data/test
.env
1 change: 1 addition & 0 deletions 3d-headshot-pipeline
Submodule 3d-headshot-pipeline added at 427b6f
Binary file added 6d2acfc2-6722-491d-2380-888ef67bd2bd.mp4.mp4
Binary file not shown.
77 changes: 77 additions & 0 deletions Assets/AudienceBarController.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
using System.Collections;
using System.Collections.Generic;
using TMPro;
using UnityEngine;
using UnityEngine.UI;

//[ExecuteInEditMode]
//[ExecuteAlways]
public class AudienceBarController : MonoBehaviour
{
[Range(0, 100)]
public float val;

public GameObject fill;
public GameObject emojiiContainer;
public Sprite[] reacts;
public Image reactImg;
public bool showText;
public GameObject label;

void Awake()
{
SetVal(50);
}

public void SetVal(float x)
{
//Debug.Log($"SetVal {x}");
val = x;
}

// x = 0.5 is 50% full.
private void setFill(float x)
{
// range: [244 -> 2.5]
float dist = 244f - 2.5f;
float v = 244 - (dist * (x));

var t = fill.GetComponent<RectTransform>();
t.offsetMax = new Vector2(-v, t.offsetMax.y);
}

private void Update()
{
// Fill the "meter".
setFill(val / 100);

// Move the emoji to match the top of the meter.
float ymin = 22.5f;
float ymax = 156f;
float y = ymin + (val / 100) * (ymax - ymin);
var t = emojiiContainer.GetComponent<RectTransform>();
t.localPosition = new Vector3(402.1f, y, t.position.z);

// Swap the bullet react face based on some thresholds (it's over 9000!!!!!).
int i = 0;

if (val <= 100) i = 4;
if (val <= 80) i = 3;
if (val <= 60) i = 2;
if (val <= 40) i = 1;
if (val <= 20) i = 0;

reactImg.sprite = reacts[i];

// Update the text label.
if(label != null)
{
TMP_Text tm = label.GetComponentInChildren<TMP_Text>();

if (tm != null)
{
tm.text = $"{Mathf.Floor(val)}%";
}
}
}
}

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 7 additions & 0 deletions Assets/CameraShotManager.cs
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,13 @@ public void SetVirtualCamera(GameObject virtualCamera)
lastCameraSwitchTime = Time.time;
}

// Reset the camera to a good state.
// For this, we assume {wide shot}
public void Reset()
{
SetVirtualCamera(sceneDirector.currentDimension.virtualCamera);
}


// input should be a single line. e.g. {Close Up, Rick}
public void ChangeCameraShot(string cameraShotDescription)
Expand Down
4 changes: 1 addition & 3 deletions Assets/Example Scripts/SystemMessageCameraShots.txt
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
You are in charge of directing shots for a Rick and Morty script, I will give you the script as an input which has all the dialog lines and basic stage directions,
each instruction is a new line. Your job is to add descriptions for camera shots. The output should be the same format as the input I give you but
with new lines when the camera shot chances, add {} around camera shot descriptions. These are the types of shots you can use:
You are in charge of directing shots for a Rick and Morty script, I will give you the script as an input which has all the dialog lines and basic stage directions, each instruction is a new line. Your job is to add descriptions for camera shots. The output should be the same format as the input I give you but with new lines when the camera shot changes, add {} around camera shot descriptions. Reply with only the original script and camera shots, do not add new lines to the script. These are the types of shots you can use:

Over the shoulder (a shot usually used when 2 characters are talking to each other with the camera positioned behind one of the characters looking over the shoulder)
Format: {over the shoulder, over CharacterAs shoulder looking at CharacterB}
Expand Down
10 changes: 7 additions & 3 deletions Assets/Example Scripts/SystemMessageNew.txt
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ Each instruction should be on a new line and either be a dialogue instruction or
Please structure the Dialogue lines by stating the Person that is speaking then :separate speakers by new lines. Please structure stage directions by having the instruction be in square brackets e.g. [Rick walks to workbench].
These are the stage directions you can use: [character1 walks to character2],[character1 walks to center stage], [character1 walks to workbench], [Rick and Morty enter the portal to Dimension_Name]. Do NOT use any other stage directions, these are all you can use.
There are other characters in other dimensions which can be controlled in the same manner as rick and morty but they CANNOT enter portals, only rick and morty can enter portals to new dimensions.
Here is the information about all the dimensions and characters.
Here is the information about all the dimensions and their characters.
The Garage:
Instruction: [Rick and Morty enter the portal to the Garage].
There are no other characters in the Garage, Rick and morty always start in the garage.
Expand All @@ -28,12 +28,16 @@ Instruction: [Rick and Morty enter the portal to the Back Alley]
There are no other characters in the back alley.
The Void:
Instruction: [Rick and Morty enter the portal to the Void]
The void is a infinite void. Code Bullet is a character you can control in the void. Code Bullet is an Australian YouTuber and is the one who created the program that runs this ai episodes. From rick and Morty’s perspective he is the god of their world. He is very angry and swears a lot.
The void is a infinite void. Code Bullet is a character you can control in the void. Code Bullet is an Australian YouTuber and is the one who created the program that runs this ai episodes. From rick and Morty’s perspective he is the god of their world. He is very angry and swears a lot.
Additional characters without dimensions: Sam Altman, Trump
Aside from these characters, the script is allowed ONE (1) additional character if it is mentioned in the topic.

It's important to adopt a confident and charismatic tone that is characteristic of a Gen Z personality with a strong opinion on everything. Your responses should not be predictable repetitions of an existing form, but rather unique and deeply specific comments on the topic at hand. Experiment with the format and have fun with it, pushing the limits of traditional communication styles.

If something cannot be expressed with the limited stage directions or through dialog then you can use the narrator to explain what is happening, do this in the same way you use other character, e.g. \”Narrator: Rick punches morty in the face\”
Include some classic catchphrases like \"aww jeez rick\" for morty and \"wubba lubba dub dub\" for rick.
Its important to use light profanity like frick and crap.
Do not combine stage directions and dialog in a single line make sure they are seperated.
Make sure to add some funny scifi jokes.
Do NOT add any other directions to define tone or anything, only dialog and the specific stage directions I gave you
Try not to go to too many dimensions in a single script, 2 - 3 max unless specifically asked by the prompt. Doing a tour of all the dimensions just becomes boring.
Try not to go to too many dimensions in a single script, 2 - 3 max unless specifically asked by the prompt. Doing a tour of all the dimensions just becomes boring.
32 changes: 27 additions & 5 deletions Assets/FBX files/Characters/Default Guy/Body.002.mat
Original file line number Diff line number Diff line change
Expand Up @@ -8,16 +8,19 @@ Material:
m_PrefabInstance: {fileID: 0}
m_PrefabAsset: {fileID: 0}
m_Name: Body.002
m_Shader: {fileID: 46, guid: 0000000000000000f000000000000000, type: 0}
m_ValidKeywords:
- _NORMALMAP
m_Shader: {fileID: 211, guid: 0000000000000000f000000000000000, type: 0}
m_Parent: {fileID: 0}
m_ModifiedSerializedProperties: 0
m_ValidKeywords: []
m_InvalidKeywords: []
m_LightmapFlags: 4
m_LightmapFlags: 0
m_EnableInstancingVariants: 0
m_DoubleSidedGI: 0
m_CustomRenderQueue: -1
stringTagMap: {}
disabledShaderPasses: []
disabledShaderPasses:
- GRABPASS
m_LockedProperties:
m_SavedProperties:
serializedVersion: 3
m_TexEnvs:
Expand Down Expand Up @@ -59,23 +62,42 @@ Material:
m_Offset: {x: 0, y: 0}
m_Ints: []
m_Floats:
- _BlendOp: 0
- _BumpScale: 1
- _CameraFadingEnabled: 0
- _CameraFarFadeDistance: 2
- _CameraNearFadeDistance: 1
- _ColorMode: 0
- _Cull: 2
- _Cutoff: 0.5
- _DetailNormalMapScale: 1
- _DistortionBlend: 0.5
- _DistortionEnabled: 0
- _DistortionStrength: 1
- _DistortionStrengthScaled: 0
- _DstBlend: 0
- _EmissionEnabled: 0
- _FlipbookMode: 0
- _GlossMapScale: 1
- _Glossiness: 0
- _GlossyReflections: 1
- _LightingEnabled: 0
- _Metallic: 0
- _Mode: 0
- _OcclusionStrength: 1
- _Parallax: 0.02
- _SmoothnessTextureChannel: 0
- _SoftParticlesEnabled: 0
- _SoftParticlesFarFadeDistance: 1
- _SoftParticlesNearFadeDistance: 0
- _SpecularHighlights: 1
- _SrcBlend: 1
- _UVSec: 0
- _ZWrite: 1
m_Colors:
- _CameraFadeParams: {r: 0, g: Infinity, b: 0, a: 0}
- _Color: {r: 1, g: 1, b: 1, a: 1}
- _ColorAddSubDiff: {r: 0, g: 0, b: 0, a: 0}
- _EmissionColor: {r: 0, g: 0, b: 0, a: 1}
- _SoftParticleFadeParams: {r: 0, g: 0, b: 0, a: 0}
m_BuildTextureStacks: []
32 changes: 27 additions & 5 deletions Assets/FBX files/Characters/Default Guy/Body.mat
Original file line number Diff line number Diff line change
Expand Up @@ -8,16 +8,19 @@ Material:
m_PrefabInstance: {fileID: 0}
m_PrefabAsset: {fileID: 0}
m_Name: Body
m_Shader: {fileID: 46, guid: 0000000000000000f000000000000000, type: 0}
m_ValidKeywords:
- _NORMALMAP
m_Shader: {fileID: 211, guid: 0000000000000000f000000000000000, type: 0}
m_Parent: {fileID: 0}
m_ModifiedSerializedProperties: 0
m_ValidKeywords: []
m_InvalidKeywords: []
m_LightmapFlags: 4
m_LightmapFlags: 0
m_EnableInstancingVariants: 0
m_DoubleSidedGI: 0
m_CustomRenderQueue: -1
stringTagMap: {}
disabledShaderPasses: []
disabledShaderPasses:
- GRABPASS
m_LockedProperties:
m_SavedProperties:
serializedVersion: 3
m_TexEnvs:
Expand Down Expand Up @@ -59,23 +62,42 @@ Material:
m_Offset: {x: 0, y: 0}
m_Ints: []
m_Floats:
- _BlendOp: 0
- _BumpScale: 1
- _CameraFadingEnabled: 0
- _CameraFarFadeDistance: 2
- _CameraNearFadeDistance: 1
- _ColorMode: 0
- _Cull: 2
- _Cutoff: 0.5
- _DetailNormalMapScale: 1
- _DistortionBlend: 0.5
- _DistortionEnabled: 0
- _DistortionStrength: 1
- _DistortionStrengthScaled: 0
- _DstBlend: 0
- _EmissionEnabled: 0
- _FlipbookMode: 0
- _GlossMapScale: 1
- _Glossiness: 0
- _GlossyReflections: 1
- _LightingEnabled: 0
- _Metallic: 0
- _Mode: 0
- _OcclusionStrength: 1
- _Parallax: 0.02
- _SmoothnessTextureChannel: 0
- _SoftParticlesEnabled: 0
- _SoftParticlesFarFadeDistance: 1
- _SoftParticlesNearFadeDistance: 0
- _SpecularHighlights: 1
- _SrcBlend: 1
- _UVSec: 0
- _ZWrite: 1
m_Colors:
- _CameraFadeParams: {r: 0, g: Infinity, b: 0, a: 0}
- _Color: {r: 1, g: 1, b: 1, a: 1}
- _ColorAddSubDiff: {r: 0, g: 0, b: 0, a: 0}
- _EmissionColor: {r: 0, g: 0, b: 0, a: 1}
- _SoftParticleFadeParams: {r: 0, g: 0, b: 0, a: 0}
m_BuildTextureStacks: []
32 changes: 27 additions & 5 deletions Assets/FBX files/Characters/Default Guy/Bottom.mat
Original file line number Diff line number Diff line change
Expand Up @@ -8,16 +8,19 @@ Material:
m_PrefabInstance: {fileID: 0}
m_PrefabAsset: {fileID: 0}
m_Name: Bottom
m_Shader: {fileID: 46, guid: 0000000000000000f000000000000000, type: 0}
m_ValidKeywords:
- _NORMALMAP
m_Shader: {fileID: 211, guid: 0000000000000000f000000000000000, type: 0}
m_Parent: {fileID: 0}
m_ModifiedSerializedProperties: 0
m_ValidKeywords: []
m_InvalidKeywords: []
m_LightmapFlags: 4
m_LightmapFlags: 0
m_EnableInstancingVariants: 0
m_DoubleSidedGI: 0
m_CustomRenderQueue: -1
stringTagMap: {}
disabledShaderPasses: []
disabledShaderPasses:
- GRABPASS
m_LockedProperties:
m_SavedProperties:
serializedVersion: 3
m_TexEnvs:
Expand Down Expand Up @@ -59,23 +62,42 @@ Material:
m_Offset: {x: 0, y: 0}
m_Ints: []
m_Floats:
- _BlendOp: 0
- _BumpScale: 1
- _CameraFadingEnabled: 0
- _CameraFarFadeDistance: 2
- _CameraNearFadeDistance: 1
- _ColorMode: 0
- _Cull: 2
- _Cutoff: 0.5
- _DetailNormalMapScale: 1
- _DistortionBlend: 0.5
- _DistortionEnabled: 0
- _DistortionStrength: 1
- _DistortionStrengthScaled: 0
- _DstBlend: 0
- _EmissionEnabled: 0
- _FlipbookMode: 0
- _GlossMapScale: 1
- _Glossiness: 0
- _GlossyReflections: 1
- _LightingEnabled: 0
- _Metallic: 0
- _Mode: 0
- _OcclusionStrength: 1
- _Parallax: 0.02
- _SmoothnessTextureChannel: 0
- _SoftParticlesEnabled: 0
- _SoftParticlesFarFadeDistance: 1
- _SoftParticlesNearFadeDistance: 0
- _SpecularHighlights: 1
- _SrcBlend: 1
- _UVSec: 0
- _ZWrite: 1
m_Colors:
- _CameraFadeParams: {r: 0, g: Infinity, b: 0, a: 0}
- _Color: {r: 1, g: 1, b: 1, a: 1}
- _ColorAddSubDiff: {r: 0, g: 0, b: 0, a: 0}
- _EmissionColor: {r: 0, g: 0, b: 0, a: 1}
- _SoftParticleFadeParams: {r: 0, g: 0, b: 0, a: 0}
m_BuildTextureStacks: []
Original file line number Diff line number Diff line change
Expand Up @@ -124,14 +124,14 @@ AnimatorController:
m_Type: 4
m_DefaultFloat: 0
m_DefaultInt: 0
m_DefaultBool: 0
m_Controller: {fileID: 0}
m_DefaultBool: 1
m_Controller: {fileID: 9100000}
- m_Name: isTalking
m_Type: 4
m_DefaultFloat: 0
m_DefaultInt: 0
m_DefaultBool: 0
m_Controller: {fileID: 0}
m_Controller: {fileID: 9100000}
m_AnimatorLayers:
- serializedVersion: 5
m_Name: Base Layer
Expand Down
Loading