Tests included in the release test suite

[MakisH]

This issue mainly serves as a discussion for adding more cases in the system tests, starting from the release_test test suite.

The test suite is specified in https://github.com/precice/tutorials/blob/develop/tools/tests/tests.yaml. We currently have:

• elastic-tube-1d
  • {fluid-cpp, solid-python} failing with different results ❌
• flow-over-heated-plate
  • {fluid-openfoam, solid-nutils} failing with different results ❌
  • {fluid-openfoam, solid-fenics} failing with different results ❌
  • {fluid-openfoam, solid-openfoam} failing with different results ❌

We should still test the following.

First priotiry

These cases are either commonly encountered by users, or they test important features.

• quickstart
• perpendicular-flap
  • {fluid-openfoam, solid-openfoam}
  • {fluid-openfoam, solid-calculix}
  • {fluid-openfoam, solid-dealii}
  • {fluid-openfoam, solid-fenics}
  • {fluid-su2, solid-calculix}
  • {fluid-nutils, solid-calculix}
• partitioned-heat-conduction
  • {openfoam-dirichlet, openfoam-neumann}
  • {fenics, fenics}
  • fenicsx ?
• flow-over-heated-plate-nearest-projection
  • {fluid-openfoam, solid-openfoam}
• heat-exchanger-simplified
  • {fluid-bottom-openfoam, solid-calculix, fluid-top-openfoam}
• multiple-perpendicular-flaps
  • {fluid-openfoam, solid-downstream-dealii, solid-upstream-dealii}
• channel-transport
  • {fluid-nutils, transport-nutils}
• flow-over-heated-plate-two-meshes
  • {fluid-openfoam, solid-calculix}
• partitioned-heat-conduction-complex
  • {fenics, fenics}
• partitioned-heat-conduction-direct
  • {nutils, nutils}
• volume-coupled-diffusion
  • {fenics, fenics}
• aste-turbine

Later

• perpendicular-flap
  • {fluid-openfoam, solid-dune}
  • {fluid-openfoam, solid-solids4foam}
  • {fluid-su2, solid-calculix}
• flow-over-heated-plate
  • {fluid-openfoam, solid-dunefem}
• channel-transport-reaction
  • {fluid-fenics, chemical-fenics}
• flow-over-heated-plate-steady-state
  • {fluid-openfoam, solid-codeaster}
• oscillator
  • {python, python}
• partitioned-elastic-beam
  • {dirichlet-calculix, neumann-calculix}

Very long cases

We don't want to test very long cases yet. The plan is to eventually restrict them to some first time steps. These are:

• elastic-tube-3d
• heat-exchanger
• turek-hron-fsi3

Cannot be tested in v2

We cannot compare reference results for cases that were not there in the previous distribution release or that had different setups:

• channel-transport
  • {fluid-openfoam, transport-nutils}
• flow-over-heated-plate-partitioned-flow
  • {fluid1-openfoam, fluid2-openfoam, solid-openfoam}
• partitioned-backwards-facing-step
  • {fluid1-openfoam, fluid2-openfoam}
• partitioned-pipe-two-phase
  • {fluid1-openfoam, fluid2-openfoam}
• partitioned-pipe (boundary conditions changed)
  • {fluid1-openfoam-pimplefoam, fluid2-openfoam-pimplefoam}
  • {fluid1-openfoam-sonicliquidfoam, fluid2-openfoam-sonicliquidfoam}
• two-scale-heat-conduction
  • {macro-nutils, micro-nutils}
• volume-coupled-flow
  • {source-nutils, fluid-openfoam}

[MakisH]

We should also make a table of test cases x components/features, to discover blind spots on what we are not testing.