Send raw bytes through data channel

[Ozzadar]

Component your feature request relates to

• Signalling server
• Frontend library
• Frontend UI library
• Matchmaker
• SFU
• This repo

Problem your feature solves? Please describe.
Currently, the only way I see to send arbitrary data through the data channel is to run everything through emitUIInteraction which runs everything throughJSON.stringify(). This makes sending arbitrary bytes through rather annoying.

Describe the solution you'd like

I'd like a function that sends a byte array ( Uint8Array.?) through the data channel without any processing.

    public emitBytes(bytes: Uint8Array) {
        if (!this._webRtcController.videoPlayer.isVideoReady()) {
            return false;
        }
        this._webRtcController.emitBytes(bytes);
        return true;
    }

Additional context

[lukehb]

Currently, the only way I see to send arbitrary data through the data channel is to run everything through emitUIInteraction which runs everything throughJSON.stringify(). This makes sending arbitrary bytes through rather annoying.

This is incorrect.

There is already an existing mechanism to send whatever payload you like over the datachannel - it is the same mechanism Pixel Streaming itself uses to send all its messages. While it is not documented, it should be quite stable in terms of APIs used.

To use It involves both sides - the frontend library and the Pixel Streaming plugin itself.

On the frontend side:

1. Use StreamMessageController, accessed via the PixelStreaming class, pixelstreaming._webRtcController.streamMessageController
2. Create a new message for the payload you want to send:

StreamMessageController.toStreamerMessages.set('MyCustomMessage', {
           id: 137, // Set an id that is not already used by Pixel Streaming, you can study this class to see which are used
           structure: ['uint8'] // Define the structure of your payload
       });

3. Send your new message using the SendMessageController, accessed via pixelstreaming._webRtcController.sendMessageController:

SendMessageController.sendMessageToStreamer("MyCustomMessage", MyPayload)

On the Pixel Streaming plugin side:

1. Use the IPixelStreamingInputHandler::RegisterMessageHandler function, accessed via

IPixelStreamingModule& Module = IPixelStreamingModule.Get();
TWeakPtr<IPixelStreamingInputHandler> InputHandler = Module.FindStreamer(Module.GetDefaultStreamerID())->GetInputHandler();

2. Bind a message handler on the UE side:

InputHandler->RegisterMessageHandler("MyCustomMessage", [](FString SourceId, FMemoryReader Ar){
    TPayloadOneParam<uint8> Payload(Ar);
   // Do something with Payload.Param1
});

If you want to see more examples of message handling, see Plugins\Media\PixelStreaming\Source\PixelStreamingInput\Private\PixelStreamingInputHandler.cpp

Additionally, if you need to send a payload of an unknown size, you can use or study the use of the string payload type.

Let me know if this is not suitable to your purposes, but this solution works today with no modification to UE, whereas the proposed method of emitBytes would require us to add new message on frontend/UE and you would have to wait until subsequent UE releases to actually use it, whereas what I have provided above works today, is extensible, and lets you define whatever payloads you want and have it handled all within the existing design of the Pixel Streaming system.

[Ozzadar]

This looks promising. Thanks, I'll give it a shot soon :)