Debugging Failing Greentea Tests

Got a Greentea test that's failing? This page will show you how to debug it and see what's going on.

Understanding the Greentea Test Runner

First thing's first: We need to go over some basics of how the Greentea test runner works. Essentially, Greentea is a system that can flash a test suite onto a device, run it, and facilitate communication with a host script. At minimum, the host script receives test results from the device over a serial port and reports success or failure. However, it can also do other things, like running USB transfers or resetting the device.

Greentea works by using a serial port to transfer information, in the form of key-value pairs, between the host and the MCU under test. When it boots, the MCU code waits until it sees a special "sync string" (which looks like {{__sync;0dad4a9d-59a3-4aec-810d-d5fb09d852c1}}) from the host. Then, it begins executing tests and reporting information to the host.

As an example in this guide, let's look at the mbed-hal-reset-reason test. This test is relatively simple, but has a host script that interacts with the embedded code. Let's say you run this test from the build directory: ctest -R mbed-hal-reset-reason (the -R argument tells CTest to only run tests which match a regex):

$ ctest -R mbed-hal-reset-reason
Test project I:/RPL/mbed-os/cmake-build-nucleo-l452re-p-greentea
    Start 13: mbed-hal-reset-reason
1/1 Test #13: mbed-hal-reset-reason ............   Passed   12.19 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =  12.44 sec

Under the hood, when you run the test, the following sequence of things happens:

1. CTest runs a small shim CMake script (<build dir>/hal/tests/TESTS/mbed_hal/reset_reason/mbed-run-greentea-mbed-hal-reset-reason.cmake) which runs the other test commands.
2. The code is built and flashed onto the device using the Mbed build system: ninja flash-mbed-hal-reset-reason. In my configuration, this calls openocd to flash the binary.
3. The MCU resets and the code boots. The code waits for the sync string from the host.
4. The test script starts mbedhtrun, which communicates with the MCU: mbedhtrun --skip-flashing -p COM3 -e I:/RPL/mbed-os/hal/tests/TESTS/mbed_hal/reset_reason/../../host_tests -m NUCLEO_L452RE_P --baud-rate=9600
5. mbedhtrun resets the device, sends the sync string, and the test starts running.
6. The device sends the message {{__host_test_name;reset_reason}}, which tells mbedhtrun to load rtc_reason.py from the host test folder I:/RPL/mbed-os/hal/tests/TESTS/mbed_hal/reset_reason/../../host_tests
7. The host test file is loaded and begins communicating with the test suite
8. The tests run and, eventually, all of them pass
9. mbedhtrun exits with a 0 exit code, indicating to CTest that the tests passed.