[WIP] Interactive - Integration with Buttplug.io

[mnh86]

New Feature / Work-in-progress

Adds support for Buttplug.io as an additional interactive device option (related to #1376) for linear axis devices.

Plan / TODOs:

• Interface to support additional interactive clients (strawman IInterface)
• Reusable FunscriptPlayer that clients can utilize to sync with video and interpolate action positions
• Implementation with buttplug-js to send position updates to Intiface Central
• SettingsInterfacePanel changes to support multiple interactive client configurations (Handy, Buttplug.io)
• Component for switching between interactive clients (ie. on ScenePlayer)
• Testing

2023 Mar 3:

• Very early work in progress branch for prototyping additional interactive clients (starting with Buttplug.io). This PR is not fully working, and not ready for review. I am currently ramping up on the codebase, seeing how the new code might fit best into the existing design, and getting a working prototype together that could be refactored later into a final maintainable state.

[mnh86]

Dev notes:

• To start simple, I'm assuming one Interactive device/client would be active at a time (ie. Handy or Buttplug.io), selected somehow by the user, perhaps from the ScenePlayer view.
• Interactive device clients might conform to an IInteractive interface which consumers like ScenePlayer use to control playback of funscripts. ie.

  interface IInteractive {
    scriptOffset: number;
    enabled(): boolean;
    connect(): Promise<void>;
    uploadScript(funscriptPath: string): Promise<void>;
    sync(): Promise<number>;
    setServerTimeOffset(offset: number): void;
    play(position: number): Promise<void>;
    pause(): Promise<void>;
    ensurePlaying(position: number): Promise<void>;
    setLooping(looping: boolean): Promise<void>;
  }
  

• The InteractiveContext.Provider has some Handy-specific behavior in it (ie. API key). Need to figure out how other individual interactive clients might be configured together from settings.

Next steps:

• Get funscript playback through buttplug.io working, quick and dirty.
• For buttplug.io, it looks to require an interval/requestAnimationFrame to handle the funscript actions, to run along side video playback, sending position data at the correct playback time. Using https://github.com/qdot/syncydink as a reference.
• Send position data to all connected ButtplugClientDevices that have linearAttributes.

[mnh86]

Dev notes: Current ScenePlayer and Interactive behavior (v0.19.1)

Play / Pause / Resume (to end):

1. Load Funscript JSON
2. Pause
3. Play position: 0 (2x?)
4. Ensure play position: 0
5. Ensure play position: 0.229491
6. Ensure play position: 0.489753
7. Ensure play position: 0.75156
8. Ensure play position: 1.013521
   ...
9. Pause
10. Play position: 1211.791061
11. Ensure play position: 1211.965281
    ...
12. Pause (at end)

Looped:

1. Load Funscript JSON
2. Pause
3. Play position: 0 (2x?)
4. Ensure play position: 0.000074
   ...
5. Ensure play position: 76.300771
6. Play position: 0
7. Ensure play position: 0.000157
   ...

Other notes:

• Changing the Video speed does not directly inform Interactive of the speed, only indirectly via "Ensure play position" updates. Could calculate new rate from sampling multiple ensurePlaying(pos) calls, but would take a momentary delay to sync whenever video speed is changed. Unsure if Handy syncs speed like this, but going to assume it does.

[mnh86]

Currently encountering issues with buttplug-js v3.1.1 sending linear commands to Intiface Central v2.3.0.

Opened buttplugio/buttplug-js#247 and buttplugio/buttplug-js#248

Will try downgrading versions to see if e2e integration can work.
Update: Tried buttplug-js 3.1.0, 3.0.0 (different errors) and Intiface v2.2.2 (same issue)

[skier233]

@mnh86 found any info out on those blockers?

[qdot]

I'll try to get the buttplug-js bug finished up in the next few days, didn't realize it was blocking this dev.

[mnh86]

I'll try to get the buttplug-js bug finished up in the next few days, didn't realize it was blocking this dev.

@qdot no worries! I have seen how busy you get from Twitter/Discord. And the earliest I'd be able to get back to this PR work is mid-April, so don't feel rushed.

[skier233]

@qdot Much appreciated!

[skier233]

@mnh86 is this unblocked now or is it still blocked by buttplug-js? cc: @qdot

[mnh86]

Going to close this PR as I won't be coming back to it for the foreseeable future.

If anyone would like to pick up where I left off, here are some things to consider:

• Try sending position updates to the buttplug.io js client using device.linear(pos, duration) syntax. The code here is sending without the duration, which causes errors on the intiface side.
• The FunscriptPlayer class would need refactor, to change from sending interpolated positions on a schedule, to sending keyframes to buttplug with durations. Buttplug will do the interpolated position movements itself and should result in smoother movement with the device.
• I felt that the InteractiveAPI component needed some sort of abstraction to support additional device interfaces. This PR worked towards that, with the IInteractive interface. But current state needs more work to fit that cleanly into SettingsInterfacePanel to configure and select one active device interface, and to fit into ScenePlayer to use an active selected device interface.