E2E testing with Testscript.

[timrulebosch]

End-to-End (E2E) testing is a testing methodology where the test focus in on the entire application workflow from a user perspective. In the context of Task, one kind of E2E testing is from the perspective of the Task CLI operation.

Tests are written in the TXTAR format and executed by the Testscript framework. The result is short, fully self-contained and concise tests which can be written by non-technical users. These test from a "contract" of expected behaviour, regardless of internal implementation.

An example test:

# Test: task init
task --init
exists Taskfile.yml
stdout 'Taskfile created: Taskfile.yml'
cmp Taskfile.yml expect


-- expect --
# https://taskfile.dev

version: '3'

vars:
  GREETING: Hello, World!

tasks:
  default:
    cmds:
      - echo "{{.GREETING}}"
    silent: true

The Testscript framework can be extended to provide custom commands/checks which can make testing more efficient. For example YAML Path queries, or more complex stdout comparison functions.

We are intending to use this method to implement acceptance tests for new Task releases, which we will run before updating our fork (of Task).

[timrulebosch]

@andreynering Hello, if you are interested in this approach please let me know and I will resolve the merge conflicts. We are interested to improve the release quality, and use a method which makes it easy to author tests. Until then, we likely develop more E2E tests with this method, and update this PR (with them).

[andreynering]

Hi @timrulebosch! Thanks for contributing to this project.

This approach looks interesting, and I may accept having this, but I'm not sure yet.

Can you further explain what do you want to have better test coverage for? As a general rule, the vast majority of the tests should be able to be written by instantiating Task "as a library" and running tests. In another words, we'd probably have very few tests that need to interact with the CLI directly.