Make mock_delay non-blocking for async acompletion calls

[gustavz]

Async acompletion runs sync completion in executor of the event loop partially with all call kwargs.
If we set mock_response to True and specify a mock_delay, then then completion does a sync time.sleep for time_delay. This blocks the event loop when making multiple acompletion calls with mock_delay concurrently.
We should instead use asyncio.sleep for the case of mocking an async response, to make sure the event loop is not blocked and concurrent calls can be mocked.

[gustavz]

OK so it seems mock_delay does indeed block async workers. We ran tests for up to 100 users and as you can see, when using mock_response together with mock_delay, total requests per seconds stay roughly flat while response times increase with increasing number of users:

whereas with real non-mocked calls, response time stays roughly flat and total requests per seconds increases with increasing number of users:

[gustavz]

@krrishdholakia for your awareness, is it expected that mock_delay blocks the async event loop and can therefore not be used for testing concurrent calls?

[gustavz]

A simple local test setup is:

import asyncio
import time
import litellm

concurrent_requests = 100

async def acompletion_with_async_sleep(mock_delay):
    await asyncio.sleep(mock_delay)
    return "pong"

async def acompletion(i):
    start_time = time.time()
    print(f"Task {i}: Starting")
    response = await litellm.acompletion(
        model="azure/gpt-4",
        messages=[{"role": "user", "content": "ping"}],
        mock_response=True,
        mock_delay=4,
        )
    print(f"Task {i}: Finished in {round(time.time() - start_time, 2)} seconds")
    return response

tasks = [acompletion(i) for i in range(concurrent_requests)]
results = await asyncio.gather(*tasks)

with output:

Task 0: Starting
Task 1: Starting
Task 2: Starting
...
Task 97: Starting
Task 98: Starting
Task 99: Starting
Task 0: Finished in 4.01 seconds
...
Task 16: Finished in 8.02 seconds
..
Task 32: Finished in 12.03 seconds
...
Task 49: Finished in 16.04 seconds
...
Task 65: Finished in 20.05 seconds
...
Task 81: Finished in 24.05 seconds
...
Task 98: Finished in 28.05 seconds

But when using acompletion_with_async_sleep instead, all tasks finish after 4 seconds