[BUG] - 500 due to Redis Timeout

[coltonbh]

Describe the bug
500 when trying to get a result.

To Reproduce
This was part of a 2000 calculation batch using CREST from the qcworkflow 5_compute.py script.

Expected behavior
Results to get returned correctly.

Additional context

10.0.1.162:39696 - "GET /api/v2/compute/output/55c2822c-aee5-492f-89fc-2d178b844555 HTTP/1.1" 500
[2025-03-07 21:15:33 +0000] [9] [ERROR] Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.13/site-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        self.scope, self.receive, self.send
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/local/lib/python3.13/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.13/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.13/site-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/usr/local/lib/python3.13/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.13/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.13/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.13/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.13/site-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.13/site-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.13/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.13/site-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.13/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.13/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.13/site-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<3 lines>...
    )
    ^
  File "/usr/local/lib/python3.13/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/chemcloud_server/routes/compute.py", line 120, in result
    prog_output.append(fr.get())
                       ~~~~~~^^
  File "/usr/local/lib/python3.13/site-packages/celery/result.py", line 247, in get
    self.maybe_throw(callback=callback)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/celery/result.py", line 365, in maybe_throw
    self.throw(value, self._to_remote_traceback(tb))
    ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/celery/result.py", line 358, in throw
    self.on_ready.throw(*args, **kwargs)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/vine/promises.py", line 235, in throw
    reraise(type(exc), exc, tb)
    ~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/vine/utils.py", line 27, in reraise
    raise value
redis.exceptions.TimeoutError: Timeout reading from socket

[coltonbh]

Happens instantaneously for this specific task id, but not others. Does not happen again upon resubmission of the same tasks. Could have sometime to do with a bad connection to the redis server? Is there a way to recycle or refresh these in the chemcloud server?

[coltonbh]

This appears to happen when the value I set to hold the task state is still in the db but the celery-task-meta has been deleted, e.g.,

In redis

c3281a9d-c338-4dd3-a930-1758e057b0be: [["c3281a9d-c338-4dd3-a930-1758e057b0be", null], null]

But no celery-task-meta-c3281a9d-c338-4dd3-a930-1758e057b0be key exists. Then when I try to run restore_result it fails with this exception. This failure occurs because an AsyncResult.get() call looks for celery-task-meta-c3281a9d-c338-4dd3-a930-1758e057b0be but it does not exist in the database.

The error is specifically raised inside the loop after future_res.ready() already said true. So it is a race condition of some sort where a task is ready and then subsequently deleted by the time we arrive at prog_output.append(fr.get()).

I think it is not possible in the client code for subsequent requests for a result to be initiated before all prior attempts have been collected (due to a new async loop starting for each run over tasks), but maybe a former attempt succeeds and then deletes a result (partially??) and then a new coroutine starts a request for the result and the delete happens between the future_res.ready() and .get() call?? Seems odd...

Not a race condition because it happens subsequently for all requests for task id in the server. That means that future_res.is_ready() was returning true on each request. Why??