Plugins modifying output can break pytester

[hseg]

When executing pytester.runpytest in an environment containing a pytest plugin that modifies the output, the result is that vanilla pytester tries and fails to parse the modified output, yielding failed assert_outcomes.

For example, the following test suite, in an environment with pytest-pretty installed:

import pytest

@pytest.mark.parametrize('params', [[], ['-p pretty'], ['-p no:pretty']])
def test_rec(testdir, params):
    testdir.makepyfile(
    """
    import pytest
    def test_noop():
        assert True is True
    """
    )
    result = testdir.runpytest(*params)
    result.assert_outcomes(passed=1)

if __name__ == '__main__':
    print('default')
    pytest.main(plugins=['pytester'])
    print('-p pretty')
    pytest.main(plugins=['pytester', 'pretty'])
    print('-p no:pretty')
    pytest.main(['-p', 'no:pretty'], plugins=['pytester'])

succeeds only when the internal testsuite is passed -p no:pretty, regardless of the flags with which pytest.main is invoked

This is an issue in certain contexts, eg kiwicom/pytest-recording#169 In that case, packaging guidelines for Arch Linux mandate running the builders without venvs. And while guidelines there mandate building packages in a chroot with only the dependencies the package needs, it is common for users to build packages outside such a chroot, and so they pick up such plugins.

One additional factor here is that diagnosing this incompatibility is difficult -- none of the plugins involved advertise this incompatibility, and the code itself doesn't suggest anything might be wrong.

Several mitigations suggest themselves, in increasing order of complexity and safety:

• Explicitly specifying the plugins to use -- this is what Gentoo does, and I'm currently writing up a proposal have proposed that Arch should do the same. This was enabled by Option to disable plugin autoload entirely #3784
• Document prominently in the pytester documentation that it is incompatible with any plugin that modifies pytest's output
• Standardize an interface for overriding the output to ensure that any fixture that calls runpytest will have a coherent set of bindings. eg enable explicitly setting which reporter pytest uses for its reporting, and then have runpytest explicitly but overridably set the reporter (so it's always parsing the report with the reporter it was expecting to be using)
• Give pytest a way to emit a machine-readable version of the log, so it's not dependent on its output not being modified (eg have a flag for machine-readable output, and use that with runpytest under the hood)

pytest version: 8.3.5

[Zac-HD]

I think we could both document this issue, and ensure that pytester raises a better error message if it fails to parse output.

[RonnyPfannschmidt]

We should probably work towards explicitly managing autoload for pytester