Skip to content

start job fails when trying to set status #1065

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
jdries opened this issue Feb 24, 2025 · 3 comments · May be fixed by #1106
Open

start job fails when trying to set status #1065

jdries opened this issue Feb 24, 2025 · 3 comments · May be fixed by #1106
Assignees
@JeroenVerstraelen

Comments

@jdries
Copy link
Contributor

jdries commented Feb 24, 2025

After starting a job and setting the application id, we also set the status.
This is of lesser importance, as status will eventually be configured by job tracker.

However, it seems that the 'start_job' call can fail on this bit. This for instance makes integrationtests less stable, but can also diminish user experience.
In the case below, the job was still started and finished successfully, so there was no real reason to return this error. Also returning the error might cause user to try and start job again, which is not a good idea when it has already been started.

Traceback (most recent call last):
  File "/opt/venv/lib64/python3.8/site-packages/urllib3/connectionpool.py", line 716, in urlopen
    httplib_response = self._make_request(
  File "/opt/venv/lib64/python3.8/site-packages/urllib3/connectionpool.py", line 468, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/opt/venv/lib64/python3.8/site-packages/urllib3/connectionpool.py", line 463, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib64/python3.8/http/client.py", line 1347, in getresponse
    response.begin()
  File "/usr/lib64/python3.8/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python3.8/http/client.py", line 276, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venv/lib64/python3.8/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
  File "/opt/venv/lib64/python3.8/site-packages/urllib3/connectionpool.py", line 802, in urlopen
    retries = retries.increment(
  File "/opt/venv/lib64/python3.8/site-packages/urllib3/util/retry.py", line 552, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/opt/venv/lib64/python3.8/site-packages/urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
  File "/opt/venv/lib64/python3.8/site-packages/urllib3/connectionpool.py", line 716, in urlopen
    httplib_response = self._make_request(
  File "/opt/venv/lib64/python3.8/site-packages/urllib3/connectionpool.py", line 468, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/opt/venv/lib64/python3.8/site-packages/urllib3/connectionpool.py", line 463, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib64/python3.8/http/client.py", line 1347, in getresponse
    response.begin()
  File "/usr/lib64/python3.8/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python3.8/http/client.py", line 276, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/jobregistry.py", line 379, in _do_request
    response = do_request()
  File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/jobregistry.py", line 363, in <lambda>
    do_request = lambda: self._session.request(
  File "/opt/venv/lib64/python3.8/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/venv/lib64/python3.8/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/opt/venv/lib64/python3.8/site-packages/requests/adapters.py", line 682, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/venv/lib64/python3.8/site-packages/flask/app.py", line 880, in full_dispatch_request
    rv = self.dispatch_request()
  File "/opt/venv/lib64/python3.8/site-packages/flask/app.py", line 865, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
  File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/users/auth.py", line 95, in decorated
    return f(*args, **kwargs)
  File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/views.py", line 1013, in start_job
    backend_implementation.batch_jobs.start_job(job_id=job_id, user=user)
  File "/opt/venv/lib64/python3.8/site-packages/openeogeotrellis/backend.py", line 1710, in start_job
    self._start_job(job_id, user, _get_vault_token,proxy_user=proxy_user)
  File "/opt/venv/lib64/python3.8/site-packages/openeogeotrellis/backend.py", line 2262, in _start_job
    dbl_registry.set_status(job_id, user_id, JOB_STATUS.QUEUED)
  File "/opt/venv/lib64/python3.8/site-packages/openeogeotrellis/job_registry.py", line 872, in set_status
    self.elastic_job_registry.set_status(job_id=job_id, status=status, started=started, finished=finished)
  File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/jobregistry.py", line 547, in set_status
    return self._update(job_id=job_id, data=data)
  File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/jobregistry.py", line 553, in _update
    return self._do_request("PATCH", f"/jobs/{job_id}", json=data)
  File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/jobregistry.py", line 382, in _do_request
    raise EjrApiError(f"Failed to do EJR API request `{method} {url}`") from e
openeo_driver.jobregistry.EjrApiError: Failed to do EJR API request `PATCH https://jobregistry.vgt.vito.be/jobs/j-2502240709194ee69e95d3c4a6bdc4fb`

https://kibana-infra.vgt.vito.be/app/kibana#/discover?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-7d,to:now))&_a=(columns:!(message),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'907d8590-4f1b-11ed-8cc4-3747d5233c59',key:req_id,negate:!f,params:(query:r-2502240709194b6380a2a158a27cf4f3),type:phrase),query:(match:(req_id:(query:r-2502240709194b6380a2a158a27cf4f3,type:phrase))))),index:'907d8590-4f1b-11ed-8cc4-3747d5233c59',interval:auto,query:(language:kuery,query:''),sort:!(!('@timestamp',desc)))
@JeroenVerstraelen
Copy link
Contributor

@jdries Is this specifically for the case where we set a job to status QUEUED? I guess we can let set_status fail silently in this specific case, the user will just see the job as created until it starts running. It's not ideal but better than the alternative where _start_job errors even though the batch job runs and finishes.

JeroenVerstraelen added a commit that referenced this issue Mar 24, 2025
@JeroenVerstraelen
fail silently when setting job to queued #1065
a510f2b
@jdries
Copy link
Contributor Author

jdries commented Mar 24, 2025

@JeroenVerstraelen yes, exactly

@JeroenVerstraelen JeroenVerstraelen linked a pull request Mar 24, 2025 that will close this issue
fail silently when setting job to queued #1065 #1106
Open
@JeroenVerstraelen
Copy link
Contributor

@jdries PR created: https://github.com/Open-EO/openeo-geopyspark-driver/pull/1106/files

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@JeroenVerstraelen JeroenVerstraelen

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fail silently when setting job to queued #1065 Open-EO/openeo-geopyspark-driver
2 participants
@JeroenVerstraelen @jdries