pixi-live2d-display

English | 中文

Live2D integration for PixiJS v7.

This project aims to be a universal Live2D framework on the web platform. While the official Live2D framework is just complex and problematic, this project has rewritten it to unify and simplify the APIs, which allows you to control the Live2D models on a high level without the need to learn how the internal system works.

Feel free to support the Maintainer:

Features

• Supports all versions of Live2D models
• Supports PIXI.RenderTexture and PIXI.Filter
• Pixi-style transform APIs: position, scale, rotation, skew, anchor
• Automatic interactions: focusing, hit-testing
• Enhanced motion reserving logic compared to the official framework
• Loading from uploaded files / zip files (experimental)
• Fully typed - we all love types!
• Live Lipsync

Requirements

• PixiJS: 7.x
• Cubism core: 2.1 or 4
• Browser: WebGL, ES6

Demos

• Basic demo
• Interaction demo
• Render texture & filter demo
• Live2D Viewer Online

Documentations

• Documentation
• API Documentation
• Development Notes

Cubism

Cubism is the name of Live2D SDK. There are so far three versions of it: Cubism 2.1, Cubism 3 and Cubism 4; where Cubism 4 is backward-compatible with Cubism 3 models.

This plugin supports all variants of Live2D models by using Cubism 2.1 and Cubism 4.

Cubism Core

Before using the plugin, you'll need to include the Cubism runtime library, aka Cubism Core.

For Cubism 4, you need live2dcubismcore.min.js that can be extracted from the Cubism 4 SDK, or be referred by a direct link (however the direct link is quite unreliable, don't use it in production!).

For Cubism 2.1, you need live2d.min.js. It's no longer downloadable from the official site since 2019/9/4, but can be found here, and with a CDN link that you'll probably need.

Individual Bundles

The plugin provides individual bundles for each Cubism version to reduce your app's size when you just want to use one of the versions.

Specifically, there are cubism2.js and cubism4.js for respective runtime, along with an index.js that includes both of them.

Note that if you want both the Cubism 2.1 and Cubism 4 support, use index.js, but not the combination of cubism2.js and cubism4.js.

To make it clear, here's how you would use these files:

• Use cubism2.js+live2d.min.js to support Cubism 2.1 models
• Use cubism4.js+live2dcubismcore.min.js to support Cubism 3 and Cubism 4 models
• Use index.js+live2d.min.js+live2dcubismcore.min.js to support all versions of models

Installation

Via npm

npm install pixi-live2d-display-lipsyncpatch

import { Live2DModel } from 'pixi-live2d-display-lipsyncpatch';
// or import { Live2DModel } from 'pixi-live2d-display-lipsyncpatch'; // i didn't test this

// if only Cubism 2.1
import { Live2DModel } from 'pixi-live2d-display-lipsyncpatch/cubism2';
// or import { Live2DModel } from 'pixi-live2d-display-lipsyncpatch/cubism2';

// if only Cubism 4
import { Live2DModel } from 'pixi-live2d-display-lipsyncpatch/cubism4';
// or import { Live2DModel } from 'pixi-live2d-display-lipsyncpatch/cubism4';

Via CDN (lipsync patched)

<!-- Load Cubism and PixiJS -->
<script src="https://cubism.live2d.com/sdk-web/cubismcore/live2dcubismcore.min.js"></script>
<script src="https://cdn.jsdelivr.net/gh/dylanNew/live2d/webgl/Live2D/lib/live2d.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/[email protected]/dist/pixi.min.js"></script>


<!-- if support for both Cubism 2.1 and 4 -->
<script src="https://cdn.jsdelivr.net/gh/RaSan147/[email protected]/dist/index.min.js"></script>

<!-- if only Cubism 2.1 support-->
<script src="https://cdn.jsdelivr.net/gh/RaSan147/[email protected]/dist/cubism2.min.js"></script>

<!-- if only Cubism 4 support-->
<script src="https://cdn.jsdelivr.net/gh/RaSan147/[email protected]/dist/cubism4.min.js"></script>

In this way, all the exported members are available under PIXI.live2d namespace, such as PIXI.live2d.Live2DModel.

Basic usage

On demand import (Recommend)

import { Live2DModel } from 'pixi-live2d-display-lipsyncpatch';
import { Application, Ticker } from 'pixi.js';

(async function () {
  const app = Application({
    view: document.getElementById('canvas'),
  });

  const model = await Live2DModel.from('shizuku.model.json', {
    // register Ticker for model
    ticker: Ticker.shared,
  });

  app.stage.addChild(model);

  // transforms
  model.x = 100;
  model.y = 100;
  model.rotation = Math.PI;
  model.skew.x = Math.PI;
  model.scale.set(2, 2);
  model.anchor.set(0.5, 0.5);

  // interaction
  model.on('hit', (hitAreas) => {
    if (hitAreas.includes('body')) {
      model.motion('tap_body');
    }
  });
})();

Full import PIXI

import * as PIXI from 'pixi.js';
import { Live2DModel } from 'pixi-live2d-display-lipsyncpatch';

// expose PIXI to window so that this plugin is able to
// reference window.PIXI.Ticker to automatically update Live2D models
window.PIXI = PIXI;

(async function () {
    const app = new PIXI.Application({
        view: document.getElementById('canvas'),
    });

    const model = await Live2DModel.from('shizuku.model.json');

    app.stage.addChild(model);

    // transforms
    model.x = 100;
    model.y = 100;
    model.rotation = Math.PI;
    model.skew.x = Math.PI;
    model.scale.set(2, 2);
    model.anchor.set(0.5, 0.5);

    // interaction
    model.on('hit', (hitAreas) => {
        if (hitAreas.includes('body')) {
            model.motion('tap_body');
        }
    });
})();

Do some motion manually

• First either you need to load your model on Live2d viewer app, or the Website by guansss here
• Check for motion category names (like "idle", "" (blank) etc)
  • Screenshot will be added soon
• Under those motion categories, each motions are used by their index
• Motion Priority table:
  • 0: no priority
  • 1: maybe [for idle animation]
  • 2: normal [default when normal action]
  • 3: Just do it! Do id now! [Forced] [default when using audio]
• Time to code

const category_name = "Idle" // name of the morion category
const animation_index = 0 // index of animation under that motion category [null => random]
const priority_number = 3 // if you want to keep the current animation going or move to new animation by force [0: no priority, 1: idle, 2: normal, 3: forced]
const audio_link = "https://cdn.jsdelivr.net/gh/RaSan147/[email protected]/playground/test.mp3" //[Optional arg, can be null or empty] [relative or full url path] [mp3 or wav file]
const volume = 1; //[Optional arg, can be null or empty] [0.0 - 1.0]
const expression = 4; //[Optional arg, can be null or empty] [index|name of expression]
const resetExpression = true; //[Optional arg, can be null or empty] [true|false] [default: true] [if true, expression will be reset to default after animation is over]

model.motion(category_name, animation_index, priority_number, {
  sound: audio_link,
  volume: volume,
  expression: expression,
  resetExpression: resetExpression
});

// if you dont want voice, just ignore the option
model.motion(category_name, animation_index, priority_number);
model.motion(category_name, animation_index, priority_number, { expression: expression, resetExpression: resetExpression });
model.motion(category_name, animation_index, priority_number, { expression: expression, resetExpression: false });

Lipsync Only

• You can do sound lipsync even without triggering any motion
• This supports expressions arg too (if you have/need any)
• Demo code

const audio_link = "https://cdn.jsdelivr.net/gh/RaSan147/[email protected]/playground/test.mp3" // [relative or full url path] [mp3 or wav file]
const volume = 1; // [Optional arg, can be null or empty] [0.0 - 1.0]
const expression = 4; // [Optional arg, can be null or empty] [index|name of expression]
const resetExpression = true; // [Optional arg, can be null or empty] [true|false] [default: true] [if true, expression will be reset to default after animation is over]
const crossOrigin = "anonymous"; // [Optional arg, to use not same-origin audios] [DEFAULT: null]

model.speak(audio_link, {
  volume: volume,
  expression: expression,
  resetExpression: resetExpression,
  crossOrigin: crossOrigin
});

// Or if you want to keep some things default
model.speak(audio_link);
model.speak(audio_link, { volume: volume });
model.speak(audio_link, { expression: expression, resetExpression: resetExpression });

Fix "MediaElementAudioSource outputs zeroes due to CORS access restrictions for"

• Both functions have crossOrigin param. Setting that to crossOrigin : "anonymous" will fix it.

model.speak(audio_link, { expression: expression, crossOrigin: "anonymous" });

model.motion(category_name, animation_index, priority_number, { sound: audio_link, volume: volume, crossOrigin: "anonymous" });

Suddenly stop audio and lipsync

• Demo code

model.stopSpeaking();

Reset motions as well as audio and lipsync

• Demo code

model.stopMotions();

Use callback function after a voiceline is over

• Demo code

model.speak(audio_link, {
  volume: volume, 
  onFinish: () => { console.log("Voiceline is over") },
  onError: (err) => { console.log("Error: ", err) } // [if any error occurs]
});

model.motion(category_name, animation_index, priority_number, {
  sound: audio_link,
  volume: volume,
  expression: expression,
  onFinish: () => { console.log("Voiceline and Animation is over") },
  onError: (err) => { console.log("Error: ", err) } // [if any error occurs]
});

Totally destroy the model

• This will also stop the motion and audio from running and hide the model
• Demo code

model.destroy();

Result

20230408191713_1.mp4

See here for more Documentation: Documentation

Package importing

When importing Pixi packages on-demand, you may need to manually register some plugins to enable optional features.

import { Application } from '@pixi/app';
import { Ticker, TickerPlugin } from '@pixi/ticker';
import { InteractionManager } from '@pixi/interaction';
import { Live2DModel } from 'pixi-live2d-display';

// register Ticker for Live2DModel
Live2DModel.registerTicker(Ticker);

// register Ticker for Application
Application.registerPlugin(TickerPlugin);

// register InteractionManager to make Live2D models interactive
Renderer.registerPlugin('interaction', InteractionManager);

(async function () {
  const app = new Application({
    view: document.getElementById('canvas'),
  });

  const model = await Live2DModel.from('shizuku.model.json');

  app.stage.addChild(model);
})();

---

The example Live2D models, Shizuku (Cubism 2.1) and Haru (Cubism 4), are redistributed under Live2D's Free Material License.

Extra credit:

• fatalgoth for making lipsync possible.
• windjackz for creating all the cool features