Converge `package:test_runner` and `package:test`?

[dcharkes]

Currently, inside the Dart SDK we use package:test_runner. Outside the Dart SDK, people use package:test.

Some of the features align:

• They can run all files in a directory that end with test.dart.

Some features don't align:

• Skips: package:test has annotations or named arguments, the Dart SDK status files.
• Expectations: package:matcher vs package:expect

Some SDK testing requirements are built in to package:test_runner and might not make sense outside the Dart SDK:

• multi tests
• comment expectations for // [cfe] and // [analyzer]
• bots/configurations
• results database

(Some of these things might actually be useful outside the Dart SDK. Multitests? Configurations to express cross-compilation once we start supporting that in standalone Dart. A serializable output for test results?)

Having these different solutions creates friction if packages are both used inside the Dart SDK and outside the Dart SDK. (For example: packages available outside the Dart SDK for users, or shared code between Dart and Flutter.)

Maybe we could reduce tech debt by trying to converge testing solutions.

It is not clear to me what the desired end state of a convergence should be, but some possibilities:

• Using package:test syntax for skips instead of status files.
• Dropping package:expect and starting to use package:matcher.
• Building package:test_runner on top of package:test with our SDK-specific requirements? (But not reinventing the wheel for shared things.)

It's likely that trying to converge testing is a lot of effort, so we might not want to prio this. But I wanted a tracking bug.

@johnmccutchan Do we have a tracking issue on the Flutter side for converging testing frameworks?

N.b. Feel free to edit this issue to add bullet points to the list above.

[dart-github-bot]

Summary: The Dart SDK uses package:test_runner for testing, while external users use package:test. This creates friction for packages used both inside and outside the SDK due to differing features and syntax. The issue proposes converging the two testing solutions to reduce tech debt and improve consistency.

[lrhn]

The langauge and core platform library tests are deliberately run using a very simplistic test framework (package:expect). That ensures that the testing framework itself doesn't depend on features that are being tested.
(And, hopefully, test_runner itself isn't too complicated either.)

It's a non-goal to change those tests to anything more complicated.
It's also not a goal to make anyone else use package:expect. If you're not testing the language or a core dart:... library, you are free to use something else than package:expect.

For packages in pkg/... of the language SDK, that are intended to be published publicly, there should be no problem using package:test for testing. That's what many of them already do. (Just not pkg/expect itself, and its friends, obviously.)

Is the problem that doing tool/test.py pkg/... will run the package's tests with package:test_runner anyway, where running dart test in the diretory would use package:test?

[dcharkes]

For packages in pkg/... of the language SDK, that are intended to be published publicly, there should be no problem using package:test for testing. That's what many of them already do.

Oh, we do? Can you point me to some of those? Then I should wire up the tests for third_party/pkg/native differently. Are these packages tested with an invocation of tools/test.py as well or with something different?

[dcharkes]

$ tools/test.py -v -n analyzer-unittest-asserts-release-mac pkg/analyzer/test/src/diagnostics/ffi_native_test

This seems to run the test without respecting package:tests annotations.

If I add

@TestOn('linux')
library;

the test still runs on MacOS.

[lrhn]

Ack, so doing tools/test.py pkg/... uses package:test_runner, not package:test.

It still uses expect and matchers from package:test, and "works" as long as you don't use any fancy features. (Anything that would work if you run the test file directly as dart run test/some_test.dart. You can run Dart tests directly, you don't need a test runner for a single test.)

Should we allow a package to be run by package:test's test runner?
Should we make pacakge:test_runner recognize test's annotations?

(Multitests are a dead end. We should get rid of all of them and remove the functionality, and only use error tests, with // [cfe] ... etc., for recognizing multiple errors in the same file.)

[athomas]

+1 to what lrhn said (especially about multitests).

At a high level, I wonder is convergence worth it for convergence's sake?

The main argument is mostly solved by using package:test:

The Having these different solutions creates friction if packages are both used inside the Dart SDK and outside the Dart SDK. (For example: packages available outside the Dart SDK for users, or shared code between Dart and Flutter.)

Using package:test directly everywhere for everything in the Dart SDK is probably a non-goal because of different requirements. That being said, package:test_runner is too bespoke and could share more code with package:test (e.g. launching browsers should not be duplicated). In general, when the Dart SDK needs something that general users wouldn't need, it should go into package:test_runner not into package:test.

So I think to actually make progress, it's probably best to zoom in on the specific frictions that arise when using package:test in the Dart SDK. E.g. if the biggest issue is @TestOn gets ignored by package:test_runner, then we could discuss how to solve that (perhaps by running dart test, perhaps by adding support to package:test_runner).