Skip to content

Flaky tests, because exceptions in tests can be ignored #156

@UncleInf

Description

@UncleInf

When running tests with cucumber/serenity runner exceptions can be ignored, resulting in runs which pass but that should have failed. To reproduce there is a repo fork with branch with a test to reproduce the issue.

Exceptions in tests do get ignored when:

  • @Step annotation is used in single scenario
  • or atleast one scenario uses @Step annotations in feature file run

Attaching a picture from 2 runs:

  • both scenarios DOES throw exception
  • when ran as a single test without @Step annotation -- test fails (picture on the left)
  • when ran as a whole feature -- tests pass even though both tests should fail (picture on the right)

image

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions