Summary report should report failed tests using {file}:{line} format instead of {file}::{test_name}

[ssbarnea]

In order to assure that the reader can easily go directly to the failed line by clicking inside the terminal we need to ensure that the output follows the format {file}:{line} ... (space after line number). This format ensures that terminal can linkify the output and open the editor at expected lines.

At this moment the editors open only the correct file, but not at desired line.

Current output:

========================================================= short test summary info =========================================================
FAILED src/test/unit/test_collections.py::test_default_options_property - AssertionError: assert {'co...

Desired output:

========================================================= short test summary info =========================================================
FAILED src/test/unit/test_collections.py:123:test_default_options_property - AssertionError: assert {'co...

One of the problems with current proposal is that current output format {file}::{test} is used as test-selector, so if we change it it will be harder to copy/paste failed test selector and run pytest only for that test.

Does anyone have an idea about how we can allow both use cases?

Based on my testing displaying something like {file}:{line}:{test_name} seems to work correctly with iterm+vscode but not with vscode own terminal, but that is due to a known bug I reported recently. I expect them to fix it soon.

This brings me back to the idea that maybe using {file}:{line}:{test_name} may prove useful as long we ensure that running pytest -k file::test still works, mainly we would likely need to ignore line numbers when using expressions.

WDYT?

[Zac-HD]

Sounds good to me - I was careful to use a similar format for Hypothesis' Phase.explain.

I'd want to design the specific format very carefully, but identifiers are still unambiguous under this proposal because an integer literal (the line number) can never be a valid Python identifier. Count me as "tentative support, inclined to merge unless the prototype looks really hairy".

[asottile]

the filename::test_name format is pytest's test id

[ssbarnea]

I think we could keep compatibility by sanitizing test id. When filename:line:test_name pattern would be passed to pytest, it will just remove the number from the string and continue.

In fact we could even make the inclusion of the line number in the output an opt-in feature, to avoid affecting users and only if it proves very popular to make it default on next major release. This approach has the advantage that we don't even need to worry about sanitizing test id.

[asottile]

what you're proposing isn't a simple output change, you're proposing changing the format of pytest's test ids which I don't think should be taken lightly

[Zac-HD]

I certainly wouldn't support changing the test ID format, that would be a huge breaking change. If there's a clean way to add the line number when printing the summary I'd consider taking that though.