Capture c++ stack traces from core dump files on linux #38600

jhaigh0 · 2025-01-14T14:57:38Z

Description of work

Adds the recovery of C++ stack trace information in the event of a crash on linux.

Launching workbench has been changed to use subprocess.Popen (rather than multiprocessing) to make use of the pre_exec_func parameter. This is passed a function which turns on core dumps for the workbench process.
The errorreports.core_dumps will need to be set for this feature to work. This is a path to the directory in which your linux system places core dump files. I couldn't find a reliable method to determine this automatically, an alternative would be to set sysctl -w kernel.core_pattern=/tmp/mantid_core_dumps in the pre-exec function. Couldn't find a python exposed way of doing this though (other than calling subprocess.run)
The core dump file is analysed using the pystack library which can recover trace backs from all active threads.
Since pystack produces a lot of output, the data is sent compressed to the error reporter.
On IDAaaS, the core dump files produced are lz4 compressed, so that case has been considered.

Further detail of work

The error reporter won't know about this change until mantidproject/errorreports#60 is merged since the C++ traces are assigned to a new variable. Either can be merged first safely.

On a built version, pystack won't recover the file names of out C++ code, but it will recover the function names and call stacks.

To test:

https://builds.mantidproject.org/job/build_packages_from_branch/1068/artifact/

conda install jhaigh0/label/core_dump_test::mantidworkbench (make sure it is 6.12.20250325.1517-py310hd83431c_0)

In your user properties file, set errorreports.core_dumps to be where core dumps end up on your system. If it's ubuntu then it's probably /var/lib/apport/coredump (On IDAaaS /var/lib/systemd/coredump and you want to set it in /etc/mantid.local.properties probably)
Launch workbench, then run Segfault to crash.
There should be some logging output that makes sense.
Once the error reporter appears, clicking on more information you should be able to see the output from pystack. There will be a lot of different threads, but one of them will have Segfault::exec()

Reviewer

Please comment on the points listed below (full description).
Your comments will be used as part of the gatekeeper process, so please comment clearly on what you have checked during your review. If changes are made to the PR during the review process then your final comment will be the most important for gatekeepers. In this comment you should make it clear why any earlier review is still valid, or confirm that all requested changes have been addressed.

Code Review

Is the code of an acceptable quality?
Does the code conform to the coding standards?
Are the unit tests small and test the class in isolation?
If there is GUI work does it follow the GUI standards?
If there are changes in the release notes then do they describe the changes appropriately?
Do the release notes conform to the release notes guide?

Functional Tests

Do changes function as described? Add comments below that describe the tests performed?
Do the changes handle unexpected situations, e.g. bad input?
Has the relevant (user and developer) documentation been added/updated?

Does everything look good? Mark the review as Approve. A member of @mantidproject/gatekeepers will take care of it.

Gatekeeper

If you need to request changes to a PR then please add a comment and set the review status to "Request changes". This will stop the PR from showing up in the list for other gatekeepers.

sf1919 · 2025-01-17T07:50:14Z

The release note structure for v6.13 has been created. You can now rebase your branch and move the release note in this PR.

github-actions · 2025-02-18T16:45:07Z

👋 Hi, @jhaigh0,

Conflicts have been detected against the base branch. Please rebase your branch against the base branch.

github-actions · 2025-03-05T16:09:07Z

👋 Hi, @jhaigh0,

Conflicts have been detected against the base branch. Please rebase your branch against the base branch.

thomashampson

Following test instruction number three here should give me a python stacktrace, but it seems to be lost now.

qt/python/mantidqt/mantidqt/dialogs/errorreports/run_pystack.py

thomashampson

I've verified that C++ stack traces are picked up on Linux, and the Python stack traces are still working.

jclarkeSTFC

This is great, but could it work on macOS as well?

jhaigh0 · 2025-04-10T13:47:41Z

This is great, but could it work on macOS as well?

I think it could work on mac too. I was focused on linux for the first version because it's what I had to test on + the majority of our users.

If you think it's as simple as replacing the if is _linux with if is_linux or is_macos then I'm happy to throw it in, we'll need to do a round of testing on macs though. Maybe best left for a second pr after we know it's working?

jclarkeSTFC · 2025-04-10T13:51:52Z

This is great, but could it work on macOS as well?

I think it could work on mac too. I was focused on linux for the first version because it's what I had to test on + the majority of our users.

If you think it's as simple as replacing the if is _linux with if is_linux or is_macos then I'm happy to throw it in, we'll need to do a round of testing on macs though. Maybe best left for a second pr after we know it's working?

Yes I think it'll work just by changing the if statements, but it can wait for a separate PR

…#38600) * add pystack to workbench and developer environments * Turn on core dumps on linux for the inner workbench process Has to switch from multiprocessing to subprocess in order to do this (needed the preexec option). Through some testing, this should preserve the desired spawn behaviour. A new module 'workbench_process.py' has been created to use with subprocess.run (can't pass it a function) * add module to get the compressed output from pystack on the latest worbench core file * add tests for new run pystack module * add cpp traces to the report sender * base64 encode data to store as a string * show pystack output in the show more details window * add release note * update existing tests * fix pystack test * add support for lz4 compressed core files, such as those on idaaas * only import lz4 if on linux * return Path type as intended * use process id to identify core dump file * move release note to 6.13 * pass missing parameter to errorreporter constructor * only launch with preexec_func when not on windows to avoid crash * add property to user properties default file * Update comment in run_pystack --------- Co-authored-by: thomashampson <thomas.hampson@stfc.ac.uk>

* Fix some cppcheck warnings * Capture c++ stack traces from core dump files on linux (#38600) * add pystack to workbench and developer environments * Turn on core dumps on linux for the inner workbench process Has to switch from multiprocessing to subprocess in order to do this (needed the preexec option). Through some testing, this should preserve the desired spawn behaviour. A new module 'workbench_process.py' has been created to use with subprocess.run (can't pass it a function) * add module to get the compressed output from pystack on the latest worbench core file * add tests for new run pystack module * add cpp traces to the report sender * base64 encode data to store as a string * show pystack output in the show more details window * add release note * update existing tests * fix pystack test * add support for lz4 compressed core files, such as those on idaaas * only import lz4 if on linux * return Path type as intended * use process id to identify core dump file * move release note to 6.13 * pass missing parameter to errorreporter constructor * only launch with preexec_func when not on windows to avoid crash * add property to user properties default file * Update comment in run_pystack --------- Co-authored-by: thomashampson <thomas.hampson@stfc.ac.uk> --------- Co-authored-by: James Clarke <james.clarke@stfc.ac.uk> Co-authored-by: Jonathan Haigh <35813666+jhaigh0@users.noreply.github.com> Co-authored-by: thomashampson <thomas.hampson@stfc.ac.uk>

jhaigh0 force-pushed the 38405_capture_cpp_stacktraces_from_core_dumps branch from e3ba601 to 8569ea0 Compare January 14, 2025 16:28

jhaigh0 added this to the Release 6.12 milestone Jan 16, 2025

jhaigh0 force-pushed the 38405_capture_cpp_stacktraces_from_core_dumps branch 2 times, most recently from b0238c0 to 83de376 Compare January 16, 2025 13:40

jclarkeSTFC modified the milestones: Release 6.12, Release 6.13 Jan 16, 2025

jhaigh0 force-pushed the 38405_capture_cpp_stacktraces_from_core_dumps branch 3 times, most recently from 272df8c to cbc3bdd Compare January 24, 2025 09:45

jhaigh0 force-pushed the 38405_capture_cpp_stacktraces_from_core_dumps branch 5 times, most recently from 8b6f084 to 2bbc225 Compare February 17, 2025 12:31

jhaigh0 added ISIS: Core Issue and pull requests at ISIS that relate to Core features and functions Epic Used for issues and PRs that are managed under the ISIS Epic Workstream labels Feb 17, 2025

github-actions bot added the Has Conflicts Used by the bot to label pull requests that have conflicts label Feb 18, 2025

github-actions bot removed the Has Conflicts Used by the bot to label pull requests that have conflicts label Feb 19, 2025

jhaigh0 marked this pull request as ready for review February 21, 2025 15:22

jhaigh0 assigned thomashampson Feb 27, 2025

sf1919 linked an issue Mar 3, 2025 that may be closed by this pull request

Enhancement to Mantid Error Reporting mantidproject/roadmap#32

Open

github-actions bot added the Has Conflicts Used by the bot to label pull requests that have conflicts label Mar 5, 2025

jhaigh0 force-pushed the 38405_capture_cpp_stacktraces_from_core_dumps branch from 885c47e to e5d9b18 Compare March 6, 2025 09:04

github-actions bot removed the Has Conflicts Used by the bot to label pull requests that have conflicts label Mar 6, 2025

thomashampson added the Extra Attention Testers and Gate keepers should pay extra attention as this affects core aspects. label Mar 14, 2025

thomashampson requested changes Mar 14, 2025

View reviewed changes

add pystack to workbench and developer environments

1b2c633

jhaigh0 added 11 commits March 24, 2025 11:29

add cpp traces to the report sender

c593e40

base64 encode data to store as a string

1e8c656

show pystack output in the show more details window

8bdec1c

add release note

c14e80a

update existing tests

6ba8c30

fix pystack test

8b7d2c0

add support for lz4 compressed core files, such as those on idaaas

5864e48

only import lz4 if on linux

2566977

return Path type as intended

8c8cc9a

use process id to identify core dump file

267ae63

move release note to 6.13

65df9e8

jhaigh0 force-pushed the 38405_capture_cpp_stacktraces_from_core_dumps branch from e5d9b18 to 65df9e8 Compare March 24, 2025 13:58

jhaigh0 added 2 commits March 25, 2025 15:16

pass missing parameter to errorreporter constructor

33ec8d0

only launch with preexec_func when not on windows to avoid crash

d422103

jhaigh0 requested a review from thomashampson March 26, 2025 11:26

add property to user properties default file

0d1f511

thomashampson reviewed Apr 8, 2025

View reviewed changes

qt/python/mantidqt/mantidqt/dialogs/errorreports/run_pystack.py Outdated Show resolved Hide resolved

Update comment in run_pystack

73cce99

thomashampson approved these changes Apr 8, 2025

View reviewed changes

sf1919 assigned jclarkeSTFC Apr 9, 2025

jclarkeSTFC requested changes Apr 10, 2025

View reviewed changes

jclarkeSTFC approved these changes Apr 10, 2025

View reviewed changes

jclarkeSTFC merged commit c579679 into main Apr 10, 2025
10 checks passed

jclarkeSTFC deleted the 38405_capture_cpp_stacktraces_from_core_dumps branch April 10, 2025 13:52

peterfpeterson mentioned this pull request Apr 15, 2025

ornl-next packaging issue #39211

Merged

jhaigh0 mentioned this pull request May 20, 2025

Add documentation for erroreports.core_dumps property #39346

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Capture c++ stack traces from core dump files on linux #38600

Capture c++ stack traces from core dump files on linux #38600

Uh oh!

jhaigh0 commented Jan 14, 2025 •

edited

Loading

Uh oh!

sf1919 commented Jan 17, 2025

Uh oh!

github-actions bot commented Feb 18, 2025

Uh oh!

github-actions bot commented Mar 5, 2025

Uh oh!

thomashampson left a comment

Uh oh!

Uh oh!

thomashampson left a comment

Uh oh!

jclarkeSTFC left a comment

Uh oh!

jhaigh0 commented Apr 10, 2025

Uh oh!

jclarkeSTFC commented Apr 10, 2025

Uh oh!

Uh oh!

Uh oh!

Capture c++ stack traces from core dump files on linux #38600

Capture c++ stack traces from core dump files on linux #38600

Uh oh!

Conversation

jhaigh0 commented Jan 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of work

Further detail of work

To test:

Reviewer

Code Review

Functional Tests

Gatekeeper

Uh oh!

sf1919 commented Jan 17, 2025

Uh oh!

github-actions bot commented Feb 18, 2025

Uh oh!

github-actions bot commented Mar 5, 2025

Uh oh!

thomashampson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

thomashampson left a comment

Choose a reason for hiding this comment

Uh oh!

jclarkeSTFC left a comment

Choose a reason for hiding this comment

Uh oh!

jhaigh0 commented Apr 10, 2025

Uh oh!

jclarkeSTFC commented Apr 10, 2025

Uh oh!

Uh oh!

Uh oh!

jhaigh0 commented Jan 14, 2025 •

edited

Loading