WebSocket support

[ambv]

Web browsers don't support communication via UDP, citing security reasons. Vendors direct developers to use WebSockets instead.

There is a case for adding WebSocket support to serialosc to allow Web apps to communicate with Monome devices. serialosc is the de facto standard for Monome devices, I'd say almost all PC and Mac users connecting their Grids and Arcs use it, because it's the "required" driver listed in the docs and it opens up support to the most end-user software. The only major platform missing at the moment is Web browsers.

Proof of concept

I wrote a WebSocket <-> UDP bridge for serialosc in Python:
https://github.com/ambv/aiotone/blob/master/aiotone/serialosc_ws.py

It works well with pymonome running on Chrome with minimal changes:
https://mastodon.social/@ambv/113906782489634159

A bridge is inferior to direct support in serialosc since it introduces latency and a point of failure, plus it's not trivial to install for others, if they even know that they need to install anything on top of serialosc when they visit a Web app. An update to serialosc that the users can upgrade to (with homebrew on the Mac) seems like a much better proposition.

Design choices

WebSocket support would use nanomsg that Norns already depends on for ws-wrapper, which is used successfully to expose Norns output in Maiden. It's a known quantity and it's been in successful use for many years now requiring no maintenance since 2021 and really only minimal maintenance since its introduction in 2018.

There are a few design decisions to make.

One daemon or two?

The first approach would be to make an additional WS daemon shipped with serialosc. This would require additional handling of the running service during installation (like launchd integration on macOS), and would present challenges with handling the USB devices non-exclusively between the two daemons.

The better approach IMO would be add an additional port (say, 12001) to the existing daemon that would handle WebSocket connections. In this world there is still only one process handling USB communication, and WebSockets only become a different transport to achieve the same result.

One WebSocket or many?

Unlike in UDP, there is a persistent connection for a WebSocket. Therefore, it might be tempting to allow a client to deal with all communication for all devices they want from that one connection. This would require slight changes of the OSC protocol.

The alternative is to do what UDP does today. The default port 12001 would be a WebSocket for discovery and handling of connections/disconnections, while separate dynamically opened ports would be WebSockets for bi-directional communication with a particular connected device. This case doesn't require any changes to the OSC protocol, although a few details would become irrelevant like passing the return host and port. The server would reply on the open WebSocket, no need to specify the return address. The docs could just specify that those values are ignored, or should be 0 0 for a WebSocket connection.

I'm leaning towards Option 2 here, because it makes it easier to adopt existing serialosc client libraries to WebSockets, making WS just a new "transport" that's an alternative for UDP, but which still handles the same serialosc protocol.

Text or binary WS frames?

My proof-of-concept bridge exclusively uses binary frames and it works great. There's human-readable introspection of the frames in Chrome Dev Tools anyway.

Rejected Alternatives

The alternative solution for Web browsers is to reimplement the serial protocol for a given Monome device and handle it via WebUSB directly. This is an inferior proposition since it requires reimplementing the protocol, as well as the killer features of serialosc like seamless handling of disconnections and reconnections, and support for multiple devices of the same type connected at the same time. Worse yet, a user of such a Web app would have to disable serialosc for the time of using the Web app. This seems like a deal breaker. It also makes it impossible to use one Monome device with a Web app and another Monome device with software that expects serialosc.

We can also wait for Grid and Arc to support class-compliant USB MIDI, but I'm not sure when this is going to be finalized, what support will historical devices have here, and how this will play with serialosc. This is a future design from my perspective, whereas WebSocket support can be added today.

Help offer

If you decide to pursue this, I'm happy to contribute the WebSocket server code to serialosc to the design of your choice.

[artfwo]

An interesting proposal indeed, I'm up for discussing this further, although I wouldn't fully exclude the possibility of these integrations utilizing WebMIDI as https://github.com/monome/iii is taking shape.

Do you think it all can run with the current design, i.e. each serialosc-device having own websocket that is discoverable via the known websocket port with the whole discovery mechanism handled by the main daemon?

What about protocols? Is raw OSC over websockets a thing already? Also, are the any latency concerns with current websocket implementations?

[ambv]

Do you think it all can run with the current design, i.e. each serialosc-device having own websocket that is discoverable via the known websocket port with the whole discovery mechanism handled by the main daemon?

Yes, that's how my prototype works.

What about protocols? Is raw OSC over websockets a thing already?

There are a few OSC to WebSocket proxies around on GitHub, but I don't think there's anything standard there.

Also, are the any latency concerns with current websocket implementations?

There's no observable latency even with my bridge in-between serialosc and the browser. It will be even faster with a direct connection.

[artfwo]

If each serialosc-device instance has a websocket, in theory you can connect to the same device from multiple apps, right? Do you think this problem could be avoided somehow?

[ambv]

It could be with a change of the protocol where as long as there is an open connection to a device WebSocket, the service discovery WebSocket does not report that device to any new connections.

Since the UDP backend currently doesn't protect against it and it doesn't seem like this is a problem reported by users, I would personally keep the protocol as-is.

[artfwo]

Hey there @ambv ! Apologies about the radio silence, I haven't forgotten about this. After giving it a bit of thought, I'd say if you'd like a natively supported serialosc connectivity to websockets, please go ahead with the implementation.

If there is enough (a few) browser apps that use websockets, I'd be happy to assist with upstreaming the code, but I'm not sure if it's worthy of doing that until there's a certain momentum. Simply because supporting the whole websockets layer is an extra load in the long term, unless it's really used, I'm not sure if it's worth the effort.

That said, it's just my opinion, and monome or other contributors may think differently. @tehn what do you think of mainlining this proposal?

[tehn]

This sounds like an interesting development, and I think we could consider it.

But first I think it's worth exploring the WS space with a simple router script (as you've done). Core infrastructure like serialosc is not an insubstantial maintenance task, so keeping the surface area as small as possible is really preferred.

Maybe what we'd be looking for is a critical mass of people interested in programing the browser with grids and arcs--- do you know of anyone else doing this? If you made a short tutorial do you think people would get into it?

Personally I'm not interested in the browser as a creative platform, but I trust there are some people out there. If that proved to be true I think we would consider adding WS support so that the intermediary router script became unnecessary.

[capogreco]

the browser is my creative platform! We exist and would love grid / arc integration options.

@ambv fwiw I have a Web Serial API implementation working here, → repo

I think the strength of this approach is its simplicity, and disabling serialosc is honestly not such a massive price for this imho, at least for the time being

[ambv]

Personally I'm not interested in the browser as a creative platform, but I trust there are some people out there. If that proved to be true I think we would consider adding WS support so that the intermediary router script became unnecessary.

That's a reasonable position to take.

I think the strength of this approach is its simplicity, and disabling serialosc is honestly not such a massive price for this imho, at least for the time being.

Impressive work with the WebSerial API! As I mentioned above, disabling serialosc only makes sense if you have just one Monome device. If you have more than one, then losing serialosc is still a pain.

[capogreco]

... disabling serialosc only makes sense if you have just one Monome device.

I don't think this is true - I am planning on connecting the grid and the arc (when it arrives) to the browser simultaneously with Web Serial API