Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crash when only_rerun is given an actual Exception type #275

Open
jamesbraza opened this issue Sep 12, 2024 · 1 comment
Open

Crash when only_rerun is given an actual Exception type #275

jamesbraza opened this issue Sep 12, 2024 · 1 comment

Comments

@jamesbraza
Copy link

jamesbraza commented Sep 12, 2024 •
edited
Loading

When using Python 3.11.10, pytest==8.3.3, and pytest-rerunfailures==14.0, and I incorrectly pass a type (and not a string) to only_rerun=[AssertionError], it leads to a pytest crash (actual CI logs):

/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.[1](https://github.com/Future-House/paper-qa/actions/runs/10838040364/job/30075372713?pr=375#step:4:1)1/site-packages/pytest_asyncio/plugin.py:208: PytestDeprecationWarning: The configuration option "asyncio_default_fixture_loop_scope" is unset.
The event loop scope for asynchronous fixtures will default to the fixture caching scope. Future versions of pytest-asyncio will default the loop scope for asynchronous fixtures to function scope. Set the default fixture loop scope explicitly in order to avoid unexpected behavior in the future. Valid fixture loop scopes are: "function", "class", "module", "package", "session"

...

  warnings.warn(PytestDeprecationWarning(_DEFAULT_FIXTURE_LOOP_SCOPE_UNSET))
============================= test session starts ==============================
platform linux -- Python 3.11.10, pytest-8.3.3, pluggy-1.5.0
rootdir: /home/runner/work/paper-qa/paper-qa
configfile: pyproject.toml
testpaths: tests
plugins: rerunfailures-1[4](https://github.com/Future-House/paper-qa/actions/runs/10838040364/job/30075372713?pr=375#step:4:5).0, asyncio-0.24.0, sugar-1.0.0, subtests-0.13.1, xdist-3.6.1, timer-1.0.0, recording-0.13.2, anyio-4.4.0
asyncio: mode=Mode.STRICT, default_loop_scope=None
created: 4/4 workers
4 workers [99 items]

..........................................................,.,...,....... [ 69%]
......................INTERNALERROR> def worker_internal_error(
INTERNALERROR>         self, node: WorkerController, formatted_error: str
INTERNALERROR>     ) -> None:
INTERNALERROR>         """
INTERNALERROR>         pytest_internalerror() was called on the worker.
INTERNALERROR>     
INTERNALERROR>         pytest_internalerror() arguments are an excinfo and an excrepr, which can't
INTERNALERROR>         be serialized, so we go with a poor man's solution of raising an exception
INTERNALERROR>         here ourselves using the formatted message.
INTERNALERROR>         """
INTERNALERROR>         self._active_nodes.remove(node)
INTERNALERROR>         try:
INTERNALERROR> >           assert False, formatted_error
INTERNALERROR> E           AssertionError: Traceback (most recent call last):
INTERNALERROR> E               File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/_pytest/main.py", line 283, in wrap_session
INTERNALERROR> E                 session.exitstatus = doit(config, session) or 0
INTERNALERROR> E                                      ^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/_pytest/main.py", line 337, in _main
INTERNALERROR> E                 config.hook.pytest_runtestloop(session=session)
INTERNALERROR> E               File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line [5](https://github.com/Future-House/paper-qa/actions/runs/10838040364/job/30075372713?pr=375#step:4:6)13, in __call__
INTERNALERROR> E                 return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR> E                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR> E                 return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> E                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 139, in _multicall
INTERNALERROR> E                 raise exception.with_traceback(exception.__traceback__)
INTERNALERROR> E               File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 122, in _multicall
INTERNALERROR> E                 teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR> E                 ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/_pytest/logging.py", line 803, in pytest_runtestloop
INTERNALERROR> E                 return (yield)  # Run all the tests.
INTERNALERROR> E                         ^^^^^
INTERNALERROR> E               File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 122, in _multicall
INTERNALERROR> E                 teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR> E                 ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/_pytest/terminal.py", line [6](https://github.com/Future-House/paper-qa/actions/runs/10838040364/job/30075372713?pr=375#step:4:7)73, in pytest_runtestloop
INTERNALERROR> E                 result = yield
INTERNALERROR> E                          ^^^^^
INTERNALERROR> E               File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 103, in _multicall
INTERNALERROR> E                 res = hook_impl.function(*args)
INTERNALERROR> E                       ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/xdist/remote.py", line 1[7](https://github.com/Future-House/paper-qa/actions/runs/10838040364/job/30075372713?pr=375#step:4:8)4, in pytest_runtestloop
INTERNALERROR> E                 self.run_one_test()
INTERNALERROR> E               File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/xdist/remote.py", line 195, in run_one_test
INTERNALERROR> E                 self.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR> E               File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 513, in __call__
INTERNALERROR> E                 return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR> E                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR> E                 return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> E                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 139, in _multicall
INTERNALERROR> E                 raise exception.with_traceback(exception.__traceback__)
INTERNALERROR> E               File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 122, in _multicall
INTERNALERROR> E                 teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR> E                 ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/_pytest/warnings.py", line 112, in pytest_runtest_protocol
INTERNALERROR> E                 return (yield)
INTERNALERROR> E                         ^^^^^
INTERNALERROR> E               File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 122, in _multicall
INTERNALERROR> E                 teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR> E                 ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/_pytest/assertion/__init__.py", line 176, in pytest_runtest_protocol
INTERNALERROR> E                 return (yield)
INTERNALERROR> E                         ^^^^^
INTERNALERROR> E               File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 122, in _multicall
INTERNALERROR> E                 teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR> E                 ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/_pytest/unittest.py", line 429, in pytest_runtest_protocol
INTERNALERROR> E                 res = yield
INTERNALERROR> E                       ^^^^^
INTERNALERROR> E               File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 122, in _multicall
INTERNALERROR> E                 teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR> E                 ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/_pytest/faulthandler.py", line [8](https://github.com/Future-House/paper-qa/actions/runs/10838040364/job/30075372713?pr=375#step:4:9)8, in pytest_runtest_protocol
INTERNALERROR> E                 return (yield)
INTERNALERROR> E                         ^^^^^
INTERNALERROR> E               File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 103, in _multicall
INTERNALERROR> E                 res = hook_impl.function(*args)
INTERNALERROR> E                       ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/pytest_rerunfailures.py", line 54[9](https://github.com/Future-House/paper-qa/actions/runs/10838040364/job/30075372713?pr=375#step:4:10), in pytest_runtest_protocol
INTERNALERROR> E                 reports = runtestprotocol(item, nextitem=nextitem, log=False)
INTERNALERROR> E                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 132, in runtestprotocol
INTERNALERROR> E                 reports.append(call_and_report(item, "call", log))
INTERNALERROR> E                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 244, in call_and_report
INTERNALERROR> E                 report: TestReport = ihook.pytest_runtest_makereport(item=item, call=call)
INTERNALERROR> E                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 513, in __call__
INTERNALERROR> E                 return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR> E                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR> E                 return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> E                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 156, in _multicall
INTERNALERROR> E                 teardown[0].send(outcome)
INTERNALERROR> E               File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/pytest_rerunfailures.py", line 517, in pytest_runtest_makereport
INTERNALERROR> E                 item._terminal_errors[result.when] = _should_hard_fail_on_error(item, result)
INTERNALERROR> E                                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/pytest_rerunfailures.py", line 286, in _should_hard_fail_on_error
INTERNALERROR> E                 return not _matches_any_rerun_error(rerun_errors, report)
INTERNALERROR> E                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/pytest_rerunfailures.py", line 255, in _matches_any_rerun_error
INTERNALERROR> E                 return _try_match_reprcrash(rerun_errors, report)
INTERNALERROR> E                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/pytest_rerunfailures.py", line 265, in _try_match_reprcrash
INTERNALERROR> E                 if re.search(rerun_regex, report.longrepr.reprcrash.message):
INTERNALERROR> E                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/home/runner/.local/share/uv/python/cpython-3.11.[10](https://github.com/Future-House/paper-qa/actions/runs/10838040364/job/30075372713?pr=375#step:4:11)-linux-x86_64-gnu/lib/python3.[11](https://github.com/Future-House/paper-qa/actions/runs/10838040364/job/30075372713?pr=375#step:4:12)/re/__init__.py", line 176, in search
INTERNALERROR> E                 return _compile(pattern, flags).search(string)
INTERNALERROR> E                        ^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/home/runner/.local/share/uv/python/cpython-3.11.10-linux-x86_64-gnu/lib/python3.11/re/__init__.py", line 286, in _compile
INTERNALERROR> E                 raise TypeError("first argument must be string or compiled pattern")
INTERNALERROR> E             TypeError: first argument must be string or compiled pattern
INTERNALERROR> E           assert False
INTERNALERROR> 
INTERNALERROR> .venv/lib/python3.11/site-packages/xdist/dsession.py:232: AssertionError
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/_pytest/main.py", line 283, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>                          ^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/_pytest/main.py", line 337, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 513, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line [12](https://github.com/Future-House/paper-qa/actions/runs/10838040364/job/30075372713?pr=375#step:4:13)0, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line [13](https://github.com/Future-House/paper-qa/actions/runs/10838040364/job/30075372713?pr=375#step:4:14)9, in _multicall
INTERNALERROR>     raise exception.with_traceback(exception.__traceback__)
INTERNALERROR>   File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 122, in _multicall
INTERNALERROR>     teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR>     ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/_pytest/logging.py", line 803, in pytest_runtestloop
INTERNALERROR>     return (yield)  # Run all the tests.
INTERNALERROR>             ^^^^^
INTERNALERROR>   File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 122, in _multicall
INTERNALERROR>     teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR>     ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/_pytest/terminal.py", line 673, in pytest_runtestloop
INTERNALERROR>     result = yield
INTERNALERROR>              ^^^^^
INTERNALERROR>   File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 103, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/xdist/dsession.py", line 138, in pytest_runtestloop
INTERNALERROR>     self.loop_once()
INTERNALERROR>   File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/xdist/dsession.py", line [16](https://github.com/Future-House/paper-qa/actions/runs/10838040364/job/30075372713?pr=375#step:4:17)3, in loop_once
INTERNALERROR>     call(**kwargs)
INTERNALERROR>   File "/home/runner/work/paper-qa/paper-qa/.venv/lib/python3.11/site-packages/xdist/dsession.py", line 2[17](https://github.com/Future-House/paper-qa/actions/runs/10838040364/job/30075372713?pr=375#step:4:18), in worker_workerfinished
INTERNALERROR>     assert not crashitem, (crashitem, node)
INTERNALERROR> AssertionError: ('tests/test_agents.py::test_successful_memory_agent', <WorkerController gw0>)
INTERNALERROR> assert not 'tests/test_agents.py::test_successful_memory_agent'

======== 91 passed, 42 warnings, 3 subtests passed in 571.[23](https://github.com/Future-House/paper-qa/actions/runs/10838040364/job/30075372713?pr=375#step:4:24)s (0:09:31) ========

Can we somehow not have a hard-to-interpret crash if this mistake is made? I am wondering if we can do something like exc.__name__ if a type is passed (instead of a str)
@icemac
Copy link
Contributor

icemac commented Sep 13, 2024

I am wondering if we can do something like exc.__name__ if a type is passed (instead of a str)

Interesting idea. A pull request is welcome.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@icemac @jamesbraza