Socketio chat, testing with pytest

[kim-andrew-v]

Hey all !

Could you guys help or explain how to get data from event handler and manipulate it. My goal is to set up testing of Chatbot on socket.io using pytest.

So i've created the client in conftest.py

import pytest
from socketio import Client
from config import LOCALHOST


sio = Client(logger=True)


@pytest.fixture(autouse=True)
def setup(client):
    client.connect(LOCALHOST)
    yield client
    client.disconnect()


@pytest.fixture
def client():
    return sio


@sio.on('bot_uttered')
def on_message(data):
    print('bot answered:', data)

In my tests i do call message as a client test.py

 def test_simple_bot(self, client):
       client.call(
           'user_uttered',
           {'message': 'Hello world!', 'session_id': client.get_sid()},
       )

But, where i'm stuck is, i can't do any assertions with response of bot which came to data in bot_uttered event handler, since it's not available under my test.

I did try to save this data to local variable, but somewhy it's not getting saved

result = ''

@sio.on('bot_uttered')
def on_message(data):
    print('bot answered:', data)
    result = data

result stays empty

same way when i do return data

@sio.on('bot_uttered')
def on_message(data):
    print('bot answered:', data)
    return data


def test_simple_bot(self, client):
      client.call(
          'user_uttered',
          {'message': 'Hello world!', 'session_id': client.get_sid()},
      )
     result = on_message
     print(result)

result is empty

Tests output

pytest e2e_tests -s
==================================== test session starts =====================================
platform darwin -- Python 3.8.10, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /Users/avkim/mts/nlp-sdk
plugins: asyncio-0.16.0, typeguard-2.13.0, cov-3.0.0
collected 1 item                                                                             

e2e_tests/test_sanity.py Engine.IO connection established
Namespace / is connected
Emitting event "user_uttered" [/]
Received event "bot_uttered" [/]
bot answered: {'text': 'Hello, i'm bot!'}
Received ack [/]
.Engine.IO connection dropped


===================================== 1 passed in 0.05s ======================================

Any suggestions?

[miguelgrinberg]

I think you just have a small bug in your logic:

result = ''

@sio.on('bot_uttered')
def on_message(data):
    nonlocal result    # <--- you have to declare result, so that the global variable is used
    print('bot answered:', data)
    result = data

[kim-andrew-v]

Hey @miguelgrinberg, thanks for the fast response!

In general you're right and it helped me to identify that the issue was in variable scope.

I ended up with client wrapper, which had all event handlers and global variable

Thank you very much!

[kim-andrew-v]

BTW, i have absolutely working solution of client, which is fits to my pytest tests

import socketio


bot_response = []


class SocketIOClient:

    client = socketio.AsyncClient(logger=True)

    async def connect(self, endpoint: Text) -> None:
        await self.client.connect(url=endpoint)

    async def disconnect(self) -> None:
        await self.client.disconnect()

    async def send_message(self, message: Text) -> None:
        await self.client.call("user_uttered", {"message": message, "session_id": self.client.get_sid()})

    @client.on('bot_uttered')
    async def on_message(data: Dict[Text, Text]) -> None:
        global bot_response
        bot_response.append(data['text'])
        print(bot_response)

    def get_response(self):
        global bot_response
        _result = bot_response
        bot_response = []
        return _result

BUT, ideally i would like to avoid using global variables. As i understand i can't pass any variable right into the handler to save data in it, because events emitted on server side, which i don't have access to (and don't need because of tests logic) and any handlers do take only one argument - data that is returned from server message event.

@miguelgrinberg Are there any workarounds to save data returned from server and avoid using global variables?

[miguelgrinberg]

There is a wait() method that you should normally use to wait for the client to complete its background tasks. I suspect the problem is that you exit immediately after calling disconnect, so you are not giving time to the background tasks to clean up and exit gracefully. Try calling await self.client.wait() right after disconnect to see if that helps.

[kim-andrew-v]

It helps! BUT not 100% times. And it greatly increases tests execution time.

Without wait():

Namespace / is connected
Emitting event "user_uttered" [/]
Received event "bot_uttered" [/]
Received event "bot_uttered" [/]
Received ack [/]
.Engine.IO connection dropped

========== 1 passed in 0.59s ========== 

with wait()

Namespace / is connected
Emitting event "user_uttered" [/]
Received event "bot_uttered" [/]
Received event "bot_uttered" [/]
Received ack [/]
.Engine.IO connection dropped


=========== 1 passed in 31.62s ===========

Sometimes failure output still exists, especially when i run bunch of tests:

Namespace / is connected
Emitting event "user_uttered" [/]
Sending packet MESSAGE data 21["user_uttered",{"message":"Hello","session_id":"WUYth9svjwoB0sfPAAA4"}]
Received packet MESSAGE data 2["bot_uttered",{"text":"Hi!"}]
Sending polling GET request to https://test.com/socket.io/?transport=polling&EIO=4&sid=sjY1_CdRoUnKQqI1AAA3
Received event "bot_uttered" [/]
Received packet MESSAGE data 2["bot_uttered",{"text":"Hi, I am your virtual assistant."}]
Received packet MESSAGE data 31[]
Sending polling GET request to https://test.com/socket.io/?transport=polling&EIO=4&sid=sjY1_CdRoUnKQqI1AAA3
Received event "bot_uttered" [/]
Received ack [/]
.Sending packet MESSAGE data 1
Sending packet CLOSE data None
Engine.IO connection dropped
Exiting write loop task
Unexpected status code 504 in server response, aborting
Waiting for write loop task to end
Exiting read loop task

And It doesn't look like 504 is the reason of the error, because it works silent when it's single test run.

[miguelgrinberg]

504 is an error that is often returned by proxy servers. Are you connecting to a proxy server in your tests?

[kim-andrew-v]

Unfortunately i'm not sure.
I think Yes, because i do use RASA framework https://rasa.com/docs/rasa/connectors/your-own-website/
And i guess it does some proxy under the hood, when it starts bot.

[miguelgrinberg]

I can't really help with this. You need to find out why this service you are using is returning an error.