Running on a remote machine over SSH with port forwarding?

[vvolhejn]

Hi, I'm trying to run the FastRTC hello world on a machine I'm SSH'd to:

import numpy as np
from fastrtc import ReplyOnPause, Stream, get_hf_turn_credentials


def echo(audio: tuple[int, np.ndarray]) -> tuple[int, np.ndarray]:
    yield audio

# When it first didn't worked, I followed the guide in https://fastrtc.org/deployment/
# and added this, but it didn't help
credentials = get_hf_turn_credentials(token=None)

stream = Stream(
    ReplyOnPause(echo),
    modality="audio",
    mode="send-receive",
    rtc_configuration=credentials,
)
stream.ui.launch()

I've forwarded the 7860 port I'm running the server on and I do see the web UI just fine. But when I press the "record" button, after a few seconds I get the "Taking a while to connect. Are you on a VPN?" popup and nothing happens.

When I run the same server locally, it works fine. In both cases, I am connected to a VPN. What could be going wrong and what is the configuration needed to make this work? Thanks!

[freddyaboulton]

Hi @vvolhejn ! Is this on the latest version of FastRTC?

The HF Turn Server is in Us-east region so it could be taking a while to resolve esp if you are using a VPN. Have you tried without VPN or with twilio?

We're working on making the HF TURN infrastructure a lot beefier in the next couple of weeks. So please bear with us 🙏

[vvolhejn]

Thanks for the response! When I run the server on localhost with the VPN active, it takes about 5 seconds to connect (also from localhost), which is likely due to what you describe. But when I run it on a remote host - let's call it remote - and then do ssh -L 7860:localhost:7860 remote to tunnel the port, and then try to connect from localhost, it seems to hang forever. Unfortunately I have no way to test that without a VPN because I need it to access remote in the first place.

Edit: Yesterday I was using version 0.0.15. I see that a new one came out in the meantime - I've re-tested with 0.0.16, with the same result.

[HsunGong]

Same problem here, I guess we need to build a custom webrtc turn server to replace the rtc_configuration. Or you need to run the client locally (only forward the vllm port).

Thanks for the response! When I run the server on localhost with the VPN active, it takes about 5 seconds to connect (also from localhost), which is likely due to what you describe. But when I run it on a remote host - let's call it remote - and then do ssh -L 7860:localhost:7860 remote to tunnel the port, and then try to connect from localhost, it seems to hang forever. Unfortunately I have no way to test that without a VPN because I need it to access remote in the first place.

Edit: Yesterday I was using version 0.0.15. I see that a new one came out in the meantime - I've re-tested with 0.0.16, with the same result.

[jjcampana]

Hi! I was experiencing a similar issue using Twilio (get_twilio_turn_credentials), and I realised that my firewall was blocking all iceServers connections to Twilio servers that were using port 3478 (UDP and TCP), waiting around 30 seconds to get a timeout to start the connection. I have fixed it only choosing the iceservers that were using port 443, that is not usually blocked by any firewall. Here you have a code snipped just in case it is useful for you:

    ice_servers = rtc_configuration["iceServers"]
    filtered_ice_servers = [server for server in ice_servers if "twilio.com:443" in server["urls"]]
    rtc_configuration["iceServers"] = filtered_ice_servers

[freddyaboulton]

I guess we need to build a custom webrtc turn server to replace the rtc_configuration

Shouldn't be necessary but please follow @jjcampana 's suggestion. The UDP/TCP ports need to be open on the remote machine.

Does this solve your issue @vvolhejn ?

[vvolhejn]

Hi! I just tried to test this. Unfortunately, Twilio doesn't allow me to use their Turn servers on a test account (no billing set):

  File ".../main.py", line 224, in <module>
    rtc_configuration = get_twilio_turn_credentials()
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../.venv/lib/python3.12/site-packages/fastrtc/credentials.py", line 38, in get_twilio_turn_credentials
    token = client.tokens.create()
            ^^^^^^^^^^^^^^^^^^^^^^
  File ".../.venv/lib/python3.12/site-packages/twilio/rest/api/v2010/account/token.py", line 102, in create
    payload = self._version.create(
              ^^^^^^^^^^^^^^^^^^^^^
  File ".../.venv/lib/python3.12/site-packages/twilio/base/version.py", line 463, in create
    return self._parse_create(method, uri, response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../.venv/lib/python3.12/site-packages/twilio/base/version.py", line 435, in _parse_create
    raise self.exception(method, uri, response, "Unable to create record")
twilio.base.exceptions.TwilioRestException:
HTTP Error Your request was:

POST /Accounts/.../Tokens.json

Twilio returned the following information:

Unable to create record: Resource not accessible with Test Account Credentials

More information may be available here:

https://www.twilio.com/docs/errors/20008

[zinccat]

same problem here, using openai api with vscode port forwarding

[grahamannett]

just chiming in to say similarly, when using remote machine cannot get fastrtc to work even with the echo back hello world type example from the blog post here: https://huggingface.co/blog/fastrtc

although using gradio directly I am able to get an audio echo back server working and it works fine if running locally. guessing there is some specific param like the rtc_configuration["iceServers"] that needs to be set on Stream init but can't find anything obvious in docs or code that seem to work.

[freddyaboulton]

Will look into this

[tungdqzenai]

hello @freddyaboulton, any update?

[Borzyszkowski]

I have the same problem when trying to run in a docker container.

Dockerfile:

# Use the official Ubuntu 20.04 image as the base
FROM ubuntu:22.04

# Set the working directory inside the container
WORKDIR /app

RUN apt-get update && apt-get install python3-pip -y
ENV DEBIAN_FRONTEND=noninteractive 

# Set the PYTHONPATH environment variable to include the `workdir` directory
ENV PYTHONPATH "${PYTHONPATH}:/app"

# Install dependencies
RUN python3 -m pip install fastrtc gradio

# Copy the current directory contents into the container
COPY . /app

Python app:

import gradio as gr
import logging

from fastrtc import Stream


def image_processor(image, slider_value):
    print(image)
    return image 


def setup_stream():
    """ Setup the stream for real-time inference """
    logging.info(f"[Job XX] Starting stream")
    stream = Stream(
        handler=image_processor,
        modality="video",
        mode="send-receive",
        additional_inputs=[
            gr.Slider(minimum=0, maximum=1, step=0.01, value=0.3)
        ],
    )
    stream.ui.launch(server_name="0.0.0.0", server_port=7865)


setup_stream()

Command to reproduce:

docker build -t my-image .
docker run -it -p 7865:7865 my-image bash
python3 example.py

I can open the window at localhost:7865 but "record" does not work correctly.

Edit: I fixed the issue by using host network for the container, instead of exposing ports:

docker run -it --network host my-image bash