switch to thread timeout method when not on main-thread #88

xoviat · 2021-04-22T21:29:50Z

This method times-out the thread on windows without crashing the test worker.

xoviat · 2021-04-27T14:10:31Z

@flub I modified this PR to do two things:

Attempt to timeout the test using the async exception method so as not to crash the worker unless absolutely necessary.
Don't use the signal method if not the main thread, avoiding pytest-xdist interop: signal only works in main thread #8.

xoviat · 2021-05-03T15:14:34Z

@flub, can you review this?

pytest_timeout.py

flub

I think one major problem you still have is that timeout_timer() unconditionally disables the capture manager and writes output. This is really not good if the thread could be interrupted by the exception since now the rest of the pytest run will behave really weirdly.

Also, there's some promising work done in #87, specifically d827511 which is exploring the same problem space. How do you think these two approaches compare?

.gitignore

pytest_timeout.py

xoviat · 2021-05-05T01:02:59Z

I've updated this to only address #8 in order to have this merged. The other bit about async timeout needs more validation, but this fix needs to be merged now to fix the bug.

.gitignore

flub · 2021-05-05T18:37:13Z

Ok, that's fair enough. Could you also add an entry to the changelog in the README for this? I think you should go ahead and bump the major version to 2.0.0 since this is a behaviour change that people should probably notice.

xoviat · 2021-05-05T22:49:43Z

@flub Done.

xoviat · 2021-05-10T16:32:45Z

@flub Anything else?

flub · 2021-05-10T19:47:43Z

@flub Anything else?

apparently some linting fixes... apologies that you didn't find this out before, stupid cryptocurrencies force me to press a silly button now before CI will run 😞

looks good otherwise!

fixes pytest-dev#8

xoviat · 2021-05-10T20:30:47Z

@flub Done.

flub · 2021-05-11T19:20:15Z

As pointed out in the other PR having a test which verifies this would be very nice. Quickly thinking about this you can probably use pytest.main() to call pytest from a non-main thread in a subprocess and check it times out.

xoviat · 2021-05-11T20:39:09Z

@flub This test may be sufficient. I declined to attempt to invoke pytest itself off the main thread because it's too difficult.

xoviat · 2021-05-13T18:42:22Z

ping @flub

ep12 · 2021-05-14T19:30:39Z

The test looks good I think. Looking forward to see this merged 👍

AZ-201 · 2022-07-19T02:58:49Z

Hello! Is this worker crash fully fixed in 2.1.0?
I am still having this problem using pytest-xdist==2.5.0 and pytest-timeout==2.1.0
env

plugins: html-3.1.1, parallel-0.1.1, metadata-1.11.0, timeout-2.1.0, forked-1.4.0, xdist-2.5.0, assume-2.4.3
timeout: 60.0s
timeout method: thread
timeout func_only: False

stacks

+++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++

~~~~~~~~~~~~~~~~~~~~~ Stack of <unknown> (139896081934080) ~~~~~~~~~~~~~~~~~~~~~
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/execnet/gateway_base.py", line 285, in _perform_spawn
    reply.run()
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/execnet/gateway_base.py", line 220, in run
    self._result = func(*args, **kwargs)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/execnet/gateway_base.py", line 967, in _thread_receiver
    msg = Message.from_io(io)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/execnet/gateway_base.py", line 432, in from_io
    header = io.read(9)  # type 1, channel 4, payload 4
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/execnet/gateway_base.py", line 400, in read
    data = self._read(numbytes - len(buf))

~~~~~~~~~~~~~~~~~~~~ Stack of MainThread (139896100013888) ~~~~~~~~~~~~~~~~~~~~~
  File "<string>", line 1, in <module>
  File "<string>", line 8, in <module>
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/execnet/gateway_base.py", line 1554, in serve
    WorkerGateway(io=io, id=id, _startcount=2).serve()
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/execnet/gateway_base.py", line 1060, in serve
    self._execpool.integrate_as_primary_thread()
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/execnet/gateway_base.py", line 267, in integrate_as_primary_thread
    self._perform_spawn(reply)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/execnet/gateway_base.py", line 285, in _perform_spawn
    reply.run()
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/execnet/gateway_base.py", line 220, in run
    self._result = func(*args, **kwargs)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/execnet/gateway_base.py", line 1084, in executetask
    do_exec(co, loc)  # noqa
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/xdist/remote.py", line 291, in <module>
    config.hook.pytest_cmdline_main(config=config)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/_pytest/main.py", line 316, in pytest_cmdline_main
    return wrap_session(config, _main)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/_pytest/main.py", line 269, in wrap_session
    session.exitstatus = doit(config, session) or 0
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/_pytest/main.py", line 323, in _main
    config.hook.pytest_runtestloop(session=session)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/xdist/remote.py", line 91, in pytest_runtestloop
    self.run_one_test(torun)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/xdist/remote.py", line 110, in run_one_test
    self.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/_pytest/runner.py", line 109, in pytest_runtest_protocol
    runtestprotocol(item, nextitem=nextitem)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/_pytest/runner.py", line 126, in runtestprotocol
    reports.append(call_and_report(item, "call", log))
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/_pytest/runner.py", line 215, in call_and_report
    call = call_runtest_hook(item, when, **kwds)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/_pytest/runner.py", line 255, in call_runtest_hook
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/_pytest/runner.py", line 311, in from_call
    result: Optional[TResult] = func()
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/_pytest/runner.py", line 255, in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/_pytest/runner.py", line 162, in pytest_runtest_call
    item.runtest()
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/_pytest/python.py", line 1641, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/_pytest/python.py", line 183, in pytest_pyfunc_call
    result = testfunction(**testargs)
  File "/home/name/test/e2e/cases/outer/test_machine.py", line 43, in test_zero
    time.sleep(60)

+++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++

[gw11] node down: Not properly terminated
[gw11] [ 99%] FAILED e2e/cases/outer/test_machine.py::Testfile::test_zero[xxxxxxxxx] 

replacing crashed worker gw11
[gw12] linux Python 3.7.3 cwd: /home/name/test                                                           
[gw12] Python 3.7.3 (default, Jan 22 2021, 20:04:44)  -- [GCC 8.3.0]                                              
gw0 [858] / gw1 [858] / gw2 [858] / gw3 [858] / gw4 [858] / gw5 [858] / gw6 [858] / gw12 [858] / gw8 [858] / gw9 [858]
e2e/cases/outer/test_machine.py::Testfile::test_1kb[xxxxxxxx] 
[gw12] [100%] PASSED e2e/cases/outer/test_machine.py::Testfile::test_1kb[xxxxxxx]

flub · 2022-07-19T18:07:14Z

@AZ-201 please submit a minimally reproducible example in a new issue if you experience a problem.

xoviat force-pushed the thread-exception branch 2 times, most recently from b000de3 to 02111fb Compare April 24, 2021 18:33

flub reviewed May 3, 2021

View reviewed changes

pytest_timeout.py Outdated Show resolved Hide resolved

pytest_timeout.py Outdated Show resolved Hide resolved

pytest_timeout.py Outdated Show resolved Hide resolved

pytest_timeout.py Outdated Show resolved Hide resolved

flub reviewed May 4, 2021

View reviewed changes

.gitignore Outdated Show resolved Hide resolved

pytest_timeout.py Outdated Show resolved Hide resolved

pytest_timeout.py Outdated Show resolved Hide resolved

ignore .vscode

c311f7a

xoviat force-pushed the thread-exception branch from 160d148 to 718682d Compare May 5, 2021 01:01

ep12 mentioned this pull request May 5, 2021

Add method = both: 1. signal + 2. thread #89

Closed

flub reviewed May 5, 2021

View reviewed changes

.gitignore Show resolved Hide resolved

use thread method if not main thread

09524aa

fixes pytest-dev#8

xoviat force-pushed the thread-exception branch from 28fc99d to 09524aa Compare May 10, 2021 20:28

flub changed the title ~~add thread-exception timeout method~~ switch to thread timeout method when not on main-thread May 11, 2021

add test to prevent regression

076b9a3

flub approved these changes May 17, 2021

View reviewed changes

flub merged commit 1a35f94 into pytest-dev:master May 17, 2021

AZ-201 mentioned this pull request Aug 29, 2022

Timeout with xdist causes worker crash and reported as error in html #137

Open

s0undt3ch mentioned this pull request Nov 22, 2023

Bad and misleading test #158

Open

switch to thread timeout method when not on main-thread #88

switch to thread timeout method when not on main-thread #88

Uh oh!

Conversation

xoviat commented Apr 22, 2021

Uh oh!

xoviat commented Apr 27, 2021

Uh oh!

xoviat commented May 3, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

flub left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xoviat commented May 5, 2021

Uh oh!

Uh oh!

flub commented May 5, 2021

Uh oh!

xoviat commented May 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xoviat commented May 10, 2021

Uh oh!

flub commented May 10, 2021

Uh oh!

xoviat commented May 10, 2021

Uh oh!

flub commented May 11, 2021

Uh oh!

xoviat commented May 11, 2021

Uh oh!

xoviat commented May 13, 2021

Uh oh!

ep12 commented May 14, 2021

Uh oh!

AZ-201 commented Jul 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

flub commented Jul 19, 2022

Uh oh!

Uh oh!

xoviat commented May 5, 2021 •

edited

Loading

AZ-201 commented Jul 19, 2022 •

edited

Loading