testthat expect* functions don't return a trace

[MichaelChirico]

Related: #2072.

I run this code:

pkgload::load_all()
reporter <- testthat::ListReporter$new()
testthat::test_local(reporter = reporter, stop_on_failure = FALSE)

And find that for any i,j such that is.null(reporter$get_results()[[ii]]$results[[j]]$trace), the test came from expect_identical(), expect_true(), ..., or other {testthat} export.

I would have expected a trace to be generated "automatically" by {testthat}'s own functions -- am I doing something wrong?

testthat_regexes <- c(
  expect_identical = "not identical to",
  expect_true_false = "is not (TRUE|FALSE)",
  expect_length = "has length [0-9]+",
  expect_null = "is not NULL",
  expect_output = "does not match.*Actual value",
  NULL
)
testthat_matcher <- paste(testthat_regexes, collapse = "|")
for (test in reporter$get_results()) {
  for (res in test$results) {
    if (inherits(res, c("expectation_success", "expectation_skip"))) next
    stopifnot("Only testthat expectations give NULL trace" =
      is.null(res$trace) == grepl(testthat_matcher, res$message))
  }
}

(NB, I have a patch applied where our main custom tester, expect_lint(), manually supplies trace=rlang::trace_back() to all 3 calls to expect())

[hadley]

Do you have any examples of failing to collect the correct trace environment materially affects testthat's reporting to the user? That would make it a bit easier for me to recollect what exactly it's a supposed to do.

[MichaelChirico]

I'm forgetting some of the context here already, but this was something I tried while writing our AST fuzz tester which works like:

1. Randomly edit tests
2. Re-run the suite
3. Examine errors, discarding those that fit into known-false-positive patterns

Here's where I wound up landing, I remember filing this bug because I found the ergonomics of my fix pretty unappealing:

https://github.com/r-lib/lintr/blob/c09fe73e0346e3df8e4dbaa81587e3401a1268d6/.dev/ast_fuzz_test.R#L102-L123

So one solution here could be to outline a better way to accomplish what I've done, or to just document that my approach works and is supported.

(to be clear, this does not affect "the user" as in a normal package author running their normal test suite, but rather a more meta-user who is programming on top of {testthat})

[hadley]

This may be off label usage of trace(), as it's mostly use to display a good trace back when a test fails. Can you back up a bit and tell me more about why you're inspecting in the output of the test reporter? I'm a bit confused because the code you link to doesn't mention trace at all.

[MichaelChirico]

Sorry again that I've lost the context...

the code you link to doesn't mention trace at all.

Right, that code represents a workaround that I landed on to approximate what I needed done.

You're right that needing trace() sounds a bit off, my memory is there was a really nice piece of detail inside the trace when it was available but the inconsistency made it unusable in the end.

Since we have something that works now, and I've lost track of the context, and it looks like there will be some overhaul to the internals here, I will close and open a new issue if I come back to this and fall into a similar trap.

Thanks!

[hadley]

Sounds good. I will also work on the docs to make sure we capture what we learn/I remember about creating expectations.