Stacktraces writing

[Kirisha]

Hello!
We want to use this helm chart, but we should patch it, because it doesn't write stacktraces.
The idea is:

1. Take PID of process in core-dump-handler (core_parten can give it)
2. Take binary file of process from path /proc/
3. Get stacktrace by corefile and binary
4. Write stacktrace to logs or send to S3

What do you think about it?

[No9]

Hey @Kirisha

Thanks for the interest in the project and excellent news that you are thinking of use cases beyond the simple capture of cores.

Not 100% what you're aiming for here. This project captures the logs from the pod and so will capture the application backtrace there?

Are you thinking about extracting the stacktrace from the core-dump when the core dump is captured in the core dump composer?
If so my current view is that this should be done with a post-processor.
There is work on going to fire events when the core dump is captured and uploaded to make down stream integrations possible.
https://github.com/IBM/core-dump-handler/tree/cloud-event - The core functionality is there but I need to finalize the best way to secure the endpoint calls - MTLS/Shared Secret or something else.
I am hesitant adding more capabilities directly at the time of capture as the capture process is outside of the kubernetes management system so unexpected peaks in usage will negatively impact the node.

All that said this is all still in investigation so I'm happy to collaborate on the best way forward

Look forward to hearing more detail on your thoughts.

[Kirisha]

Hello @No9
Thanks for the quick answer!

This project captures the logs from the pod and so will capture the application backtrace there?
I think, it will be convenient to get the stacktrace from the agent logs. Or do you mean something else here?

Are you thinking about extracting the stacktrace from the core-dump when the core dump is captured in the core dump composer?
Yes, I think it is the only way to get exe file for stacktrace writnig. I coldn't understand, how do you get exe file from agent in cloud-event?

[No9]

Hey @Kirisha
I think I understand your underlying point now.
You want to know how you can get the related exe in order to be able to get stacktraces or debug the core dump?

Again I think this is a post processing activity.
The point of capturing the pod meta data as part of the core dump is to enable the matching of the core dump back to the original container and from there get the original exe.
Just getting the backtrace is not enough in some scenarios as you need access to variables to understand why something has happened.

I was working on a cli to do this https://github.com/IBM/core-dump-client#usage but the overall UX was a bit clunky but could be improved with a serverside service for it to query so you didn't have to download the core and find the image name then run a command.

There was also a spike where I looked at consuming an Object Storage Event to create a PR that pointed to an preconfigured development environment.
https://venshare.com/blog/gitops-coredump
The cloud event idea was something that fell out of the spike as some organisations might want do this matching "on-box" before it goes to object storage in order to remove sensitive data while other organisations may not want to tie in to a specific platforms object storage event. Hence the two options.

Hopefully there are some ideas on how you can achieve what you are looking for. If you want to work more on the cli I am happy to collaborate but it's not my priority focus at the moment. Otherwise you might want to bake something yourself either from the storage event or the cloud event.