SocketIO and Gevent - trouble sending two emits asyncronously from one function

[reuip]

Hey everyone,

I have a function that is triggered by a socket.on event which starts two tasks asyncronously. That part of the code works, I can see that task1 and task2 are started in parallel. However, my goal is to have the first task emit it's result (result1) as soon as it is ready, then the second emit once result2 is in. Currently, my console only shows any change once both task1 and task2 are complete. So it seems like the emit's are blocked or waiting for the function to finish in order to send.

I have at the top of my init file the monkey patch, in case that's relevant.

from gevent import monkey monkey.patch_all()

A simplified version of my code looks like this:

def example_function(value):
     g1 = gevent.spawn(task1, value)
     g2 = gevent.spawn(task2, value)
     result1 = g1.get()
     socketio.emit( "task1", {"data" : result1} )
     result2 = g2.get()
     socketio.emit( "task2",{"data" : result2} )

On the html side:

socket.on('task1', function( response ){
            console.log( response.data );
        });

socket.on('task2', function( response ){
            console.log( response.data );
        });

I appreciate any thoughts or ideas you've got for me!

[miguelgrinberg]

I think it depends on what you do in your tasks. If your first task performs blocking or CPU intensive work, then the second task would not have a chance to run and do its thing in parallel.

[reuip]

Thanks Miguel for your super fast response.

I added print's to task1 and task2 to be able to track them in my terminal. Now I can see when they start and end with timestamps.

They start at the same time which was what I aimed for. Then, task 1 ends first. However, when it ends, I assumed the emit would send and a console message would appear.

However, instead, the console does not show any change in the log until after the second emit.

So essentially what I am seeing is (with timestamps):

• Print task1 started 20:55:53
• Print task 2 started 20:55:53
• Print task 1 done 20:57:01
• Print task 2 done 20:58:32
• Console log shows 2 messages (both task1 and task2, the results of the 2 emits) 20:58:33

What I had hoped to happen instead (with timestamps):

• Print task 1 started 20:55:53
• Print task 2 started 20:55:53
• Print task 1 done 20:57:01
• Console log shows message from 1st emit 20:57:02
• Print task 2 done 20:58:32
• Console log shows message from 2nd emit 20:58:33

Hope that clarifies a bit.

[miguelgrinberg]

This does not really give me any insight regarding what you do in your tasks. The most logical answer is still that you having something blocking or CPU intensive in your first task that prevents task from running until task 1 ends.

[reuip]

Both of my tasks are API calls, which I didn't think would be blocking. In the end, what worked was switching from gevent to concurrent.futures. Now each emit triggers when it's corresponding task is done, rather than both at the end. Hope it's helpful in case someone else has a similar issue!

So the new code looks like:

def example_function(value):
     with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
          # Run both tasks
          g1 = executor.submit(task1, value)
          g2 = executor.submit(task2, value)
         
         # Wait, blocks until tasks in list are ready (task 1 always finishes first)
         # Then emit result
         concurrent.futures.wait( [g1] )
         result1 = g1.result()
         socketio.emit( "task1", {"data" : result1} )
         
         # Wait for task2 to complete and emit result
         concurrent.futures.wait( [g2] ) 
         result2 = g2.result()     
         socketio.emit( "task2",{"data" : result2} )