Live2D with Lipsync (using audio file/link)

.github/workflows/test.yml

@@ -16,7 +16,7 @@ jobs:
   test:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
         with:
           submodules: true
 
@@ -57,3 +57,9 @@ jobs:
           name: diff-images
           path: test/**/__image_snapshots__/__diff_output__/*
           if-no-files-found: ignore
+
+
+      - uses: actions/upload-artifact@v3
+        with:
+          name: my-artifact
+          path: ${{ github.workspace }}/dist

.gitignore

@@ -7,9 +7,9 @@ core
 /build
 /*.build
 /lib
-/dist
-/types
 /site
+/types
+/dist
 
 .DS_Store
 node_modules

DEVELOPMENT.md

@@ -72,7 +72,12 @@ Or run the tests and update snapshots:
 npm run test:u
 ```
 
-If you get an error like `This version of ChromeDriver only supports Chrome version XX`, you need to either upgrade or downgrade your Chrome browser to match that version, or run `npm install chromedriver@<version>` to install the correct version of ChromeDriver (don't commit this change if you are contributing to this project!).
+If you get an error like `This version of ChromeDriver only supports Chrome version XX`, you need to do either of the following:
+
+- Check if your Chrome browser has a "Relaunch to update" button in the top right corner. If it does, click the button to update Chrome to the latest version.
+- Delete the installed `chromedriver` folder (for example `C:\Users\XXX\AppData\Local\Temp\chromedriver\win64-120.0.6099.109\chromedriver-win64` on Windows).
+
+If you get an error like `Error: spawn C:\Users\XXX\AppData\Local\Temp\chromedriver\win64-120.0.6099.109\chromedriver-win64\chromedriver.exe ENOENT`, you need to delete the `chromedriver` folder just like the step 2 above.
 
 ## Playground
 

README.md

@@ -21,10 +21,11 @@ Live2D models on a high level without the need to learn how the internal system
 - Enhanced motion reserving logic compared to the official framework
 - Loading from uploaded files / zip files (experimental)
 - Fully typed - we all love types!
+- Live Lipsync
 
 #### Requirements
 
-- PixiJS: 6.x
+- PixiJS: 7.x
 - Cubism core: 2.1 or 4
 - Browser: WebGL, ES6
 
@@ -97,9 +98,17 @@ import { Live2DModel } from 'pixi-live2d-display/cubism2';
 import { Live2DModel } from 'pixi-live2d-display/cubism4';
 ```
 
-#### Via CDN
+#### Via CDN (lipsync patched)
 
 ```html
+
+<!-- Load Cubism and PixiJS -->
+<script src="https://cubism.live2d.com/sdk-web/cubismcore/live2dcubismcore.min.js"></script>
+<script src="https://cdn.jsdelivr.net/gh/dylanNew/live2d/webgl/Live2D/lib/live2d.min.js"></script>
+<script src="https://pixijs.download/v7.4.0/pixi.min.js"></script>
+
+
+<!-- if support for both Cubism 2.1 and 4 -->
 <script src="https://cdn.jsdelivr.net/npm/pixi-live2d-display/dist/index.min.js"></script>
 
 <!-- if only Cubism 2.1 -->
@@ -122,31 +131,106 @@ import { Live2DModel } from 'pixi-live2d-display';
 window.PIXI = PIXI;
 
 (async function () {
-  const app = new PIXI.Application({
-    view: document.getElementById('canvas'),
-  });
+    const app = new PIXI.Application({
+        view: document.getElementById('canvas'),
+    });
+
+    const model = await Live2DModel.from('shizuku.model.json');
+
+    app.stage.addChild(model);
+
+    // transforms
+    model.x = 100;
+    model.y = 100;
+    model.rotation = Math.PI;
+    model.skew.x = Math.PI;
+    model.scale.set(2, 2);
+    model.anchor.set(0.5, 0.5);
+
+    // interaction
+    model.on('hit', (hitAreas) => {
+        if (hitAreas.includes('body')) {
+            model.motion('tap_body');
+        }
+    });
+})();
+```
 
-  const model = await Live2DModel.from('shizuku.model.json');
+## Do some motion manually
+* First either you need to load your model on Live2d viewer app, or the Website by guansss [here](https://guansss.github.io/live2d-viewer-web/)
+* Check for motion category names (like "idle", "" (blank) etc)
+  * Screenshot will be added soon
+* Under those motion categories, each motions are used by their index
+* Motion Priority table:
+  * 0: no priority
+  * 1: maybe [for idle animation]
+  * 2: normal [default when normal action]
+  * 3: Just do it! Do id now! [Forced] [default when using audio]
+* Time to code
+```js
+var category_name = "Idle" // name of the morion category
+var animation_index = 0 // index of animation under that motion category [null => random]
+var priority_number = 3 // if you want to keep the current animation going or move to new animation by force [0: no priority, 1: idle, 2: normal, 3: forced]
+var audio_link = "https://audio-samples.github.io/samples/mp3/blizzard_primed/sample-4.mp3" //[Optional arg, can be null or empty] [relative or full url path] [mp3 or wav file]
+var volume = 1; //[Optional arg, can be null or empty] [0.0 - 1.0]
+var expression = 4; //[Optional arg, can be null or empty] [index|name of expression]
+var resetExpression = true; //[Optional arg, can be null or empty] [true|false] [default: true] [if true, expression will be reset to default after animation is over]
+
+model.motion(category_name, animation_index, priority_number, {sound: audio_link, volume: volume, expression:expression, resetExpression:resetExpression})
+// Note: during this animation with sound, other animation will be ignored, even its forced. Once over, it'll be back to normal
+
+// if you dont want voice, just ignore the option
+model.motion(category_name, animation_index, priority_number)
+model.motion(category_name, animation_index, priority_number, {expression:expression, resetExpression:resetExpression})
+model.motion(category_name, animation_index, priority_number, {expression:expression, resetExpression:false})
 
-  app.stage.addChild(model);
+```
 
-  // transforms
-  model.x = 100;
-  model.y = 100;
-  model.rotation = Math.PI;
-  model.skew.x = Math.PI;
-  model.scale.set(2, 2);
-  model.anchor.set(0.5, 0.5);
-
-  // interaction
-  model.on('hit', (hitAreas) => {
-    if (hitAreas.includes('body')) {
-      model.motion('tap_body');
-    }
-  });
-})();
+## Lipsync Only
+* You can do sound lipsync even without triggering any motion
+* This supports expressions arg too (if you have/need any)
+* Demo code
+```js
+var audio_link = "https://audio-samples.github.io/samples/mp3/blizzard_primed/sample-4.mp3" // [relative or full url path] [mp3 or wav file]
+var volume = 1; // [Optional arg, can be null or empty] [0.0 - 1.0]
+var expression = 4; // [Optional arg, can be null or empty] [index|name of expression]
+var resetExpression = true; // [Optional arg, can be null or empty] [true|false] [default: true] [if true, expression will be reset to default after animation is over]
+
+model.speak(audio_link, {volume: volume, expression:expression, resetExpression:resetExpression})
+
+// Or if you want to keep some things default
+model.speak(audio_link)
+model.speak(audio_link, {volume: volume})
+model.speak(audio_link, {expression:expression, resetExpression:resetExpression})
+
+```
+
+## Suddenly stop audio and lipsync
+* Demo code
+```js
+model.stopSpeaking()
+```
+
+## Reset motions as well as audio and lipsync
+* Demo code
+```js
+model.stopMotions()
 ```
 
+## Totally destroy the model
+* This will also stop the motion and audio from running and hide the model
+* Demo code
+```js
+model.destroy()
+```
+
+## Result
+https://user-images.githubusercontent.com/34002411/230723497-612146b1-5593-4dfa-911d-accb331c5b9b.mp4
+
+# See here for more Documentation: [Documentation](https://guansss.github.io/pixi-live2d-display/)
+
+
+
 ## Package importing
 
 When importing Pixi packages on-demand, you may need to manually register some plugins to enable optional features.