Listener: A Novel Method for Implementing StageLinq

[honusz]

Hi everyone, I'm very excited to share something I've been working on for a while. I'm curious to know what you all think.

https://github.com/honusz/StageLinq/tree/Listener

StageLinq - Listener Proof of Concept

Description

This branch demonstrates a novel way of handling connections with devices and their services on the StageLinq network.
Rather than searching out devices via discovery, we are able to have devices initiate connections to the library. As demonstrated, this approach:

• Greatly reduces complexity.

• Speeds up the connection & initialization process (almost every sleep() call has been eliminated without and affect thus far).

• Handles disconnection and reconnection of devices gracefully and simply.

• Allows connections from devices we couldn't use previously (i.e. x1800/x1850 mixers).

Method Summary

• Instantiate a net.Server for each service we wish to implement.

• Write a DiscoveryMessage which includes the port on which the Discovery service is listening, and announce ourselves on UDP port 51337.

• StageLinq devices on the network will initiate a connection to the Directory service, and send a Service Request (0x2).

• Reply to each device with a Service Announcement (0x0) for each of the services we offer and the port we are listening to them on.

• If a device implements that service it will initiate a connection, sending a Service Announcement (0x0), a Network String with the name of the service, and the port it is using.

• The connection to this Device-Service is now open, and we can use it as we normally would.

• If a connection is lost to a device, it will simply reconnect.

Additional Notes on the Listener Method

• The Directory service is the only one which is required as it is the initial connection endpoint for remote devices.

• Only tokens of a specific structure seem to work, otherwise devices won't initiate a connection. One requirement seems to be that they start with 0xFF FF FF FF FF FF, but some more research into this is needed.

Implementing Selected Services

We can choose which services to implement in the initialize() method in StageLinqDevices.ts

  async initialize(): Promise<AddressInfo> {
    
    await this.startServiceListener(StateMap);
    await this.startServiceListener(FileTransfer);
    const directory = await this.startServiceListener(Directory); //we need the server's port for announcement message

    return directory.serverInfo
  } 

Other Changes Implemented in Listener

• When a device is shutdown, it sends a Disconnection message (0x9) to connected FileTransfer service.

• Network Device has been eliminated.

  • Directory is now an extended instance of Service.
  • The connectToService factory function has been moved to StageLinqDevices (renamed startServiceListener);

• Created a DeviceId class, to assist handling deviceIds.

  • An instance of DeviceId holds both the buffer and formatted UUID-type string.
  • It has methods for retrieving the string or buffer.

• Created some other type definitions (IpAddressPort) to better communicate what properties are expecting.

• Instances of Service have a new abstract method, parseServiceData. This is just because ServiceAnnouncement messages on StateMap and FileTransfer were causing some headache, and needed to be parsed separately.

• FileTransfer Unknown0 message is used to trigger getSources. See notes in code.

To Be Implemented / Possibilities

• EventEmitter stuff isn't fully implemented yet, with a few exceptions (those needed for FileTransfer)

• FileTransfer has only been tested with DB.

• Some methods from Database.ts have been moved to FileTransfer. This isn't intended to be permanent, it was just to allow a demonstration of downloading a DB.

• TimeSyc.ts is in the project, but should be disregarded for now.

• We could possibly eliminate StageLinqDevices, as this step is no longer really necessary. Multiple devices are handled by single instances of each service.

• StageLinqListener and Announce could be reworked into a single class. Presently all StageLinqListener is doing is adding and updating a list of announced devices (StageLinqDevices.peers).

Compatibility Notes

• This is Experimental. Do not use in a live setup, lots of further testing is required.
• That being said, I have tested it (Mac & PC) on my setup:
  • 2x SC6000 players running EngineOS 2.3.2
  • X1800 Mixer running 1.6 firmware
• Hoping Prime4 / Prime GO users could test and let me know how it works.

@MarByteBeep @icedream @chrisle @mhite @dzelionis

[chrisle]

@honusz AMAZING dude!!! When I get some free cycles this month, I'm definitely gonna dive into this and see how it works!

[honusz]

@chrisle I forgot to mention, the mixers seem to need timestamp replies to be sent, or they disconnect after not receiving a reply after like 10 or so messages.

Conscious that this was an issue, I set it to only send timestamp replies to certain software, and I set a delay replying of 1.4 seconds (the longest I could do, I think the mixer sends a new message every 1.5 seconds).
Hopefully this will prevent any jamming of other devices, but it's something we should keep in mind.

[dzelionis]

@honusz, really good job dude..! :D
I think everyone in our not official dev team is doing amazing work!
So let's keep it up!!!
@MarByteBeep @icedream @chrisle @mhite

[mhite]

@honusz - This is so awesome! I hope to one day help get the go-stagelinq library providing Stagelinq services also -- especially if you find out we can do interesting stuff like act as a remote source for a player! :)

[honusz]

@mhite the principles should be pretty easily transferable to go-stagelinq, and it's a pretty easy way to handle multiple devices (Go users scoff at Node concurrency woes, ha).

As I mentioned to you, my initial goal with this rewrite was to make experimentation a bit easier, hoping to allow this to be a good base for that.

One neat thing I discovered is you can play with the ports you send to other StageLinq devices somewhat.
You can advertise all services on the same port, and each service from the player will try to connect to that one port.

You could also split instantiate a new set of servers for each connection, and just direct them appropriately when you send your ServiceAnnouncements. The players don't do much thinking, they just try to connect where they're told.

especially if you find out we can do interesting stuff like act as a remote source for a player! :)

That totally should be possible. We could potentially serve up the Engine DJ library, eliminating USBs all together. I'm sure there's a good reason Engine doesn't do this though 😂 Maybe sync the library from Engine DJ to a NAS, run a small container applet on it to serve up the source.
Lots of cool possibilities!

[mhite]

especially if you find out we can do interesting stuff like act as a remote source for a player! :)

That totally should be possible. We could potentially serve up the Engine DJ library, eliminating USBs all together. I'm sure there's a good reason Engine doesn't do this though 😂 Maybe sync the library from Engine DJ to a NAS, run a small container applet on it to serve up the source. Lots of cool possibilities!

@honusz - You can browse USB sources that haven't been analyzed, so I figure you could also do the same sort of browsing of a remote file system over the FileTransfer protocol without even having to worry about generating a database for the files. Just a guess?