diff --git a/doc/19-technical-concepts.md b/doc/19-technical-concepts.md index de68bdd632..2d25602d8e 100644 --- a/doc/19-technical-concepts.md +++ b/doc/19-technical-concepts.md @@ -204,7 +204,7 @@ You can read the full story [here](https://github.com/Icinga/icinga2/issues/7309 With 2.11 you'll now see 3 processes: -- The umbrella process which takes care about signal handling and process spawning/stopping +- The umbrella process which takes care of signal handling and process spawning/stopping - The main process with the check scheduler, notifications, etc. - The execution helper process diff --git a/doc/21-development.md b/doc/21-development.md index cceaece200..2f17c8e835 100644 --- a/doc/21-development.md +++ b/doc/21-development.md @@ -267,73 +267,130 @@ $3 = std::vector of length 11, capacity 16 = {{static NPos = 1844674407370955161 ### Core Dump -When the Icinga 2 daemon crashes with a `SIGSEGV` signal -a core dump file should be written. This will help -developers to analyze and fix the problem. +When the Icinga 2 daemon is terminated by `SIGSEGV` or `SIGABRT`, a core dump file +should be written. This will help developers to analyze and fix the problem. -#### Core Dump File Size Limit +#### Core Dump Kernel Pattern -This requires setting the core dump file size to `unlimited`. +Core dumps are generated according to the format specified in +`/proc/sys/kernel/core_pattern`. This can either be a path relative to the +directory the program was started in, an absolute path or a pipe to a different +program. +For more information see the [core(5)](https://man7.org/linux/man-pages/man5/core.5.html) man page. -##### Systemd +#### Systemd Coredumpctl +Most distributions offer systemd's coredumpctl either by default or as a package. +Distributions that offer it by default include RHEL and SLES, on others like +Debian or Ubuntu it can be installed via the `systemd-coredump` package. +When set up correctly, `core_pattern` will look something like this: +``` +# cat /proc/sys/kernel/core_pattern +|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h` ``` -systemctl edit icinga2.service -[Service] -... -LimitCORE=infinity +You can look at the generated core dumps with the `coredumpctl list` command. +You can show information, including a stack trace using +`coredumpctl show icinga2 -1` and retrieve the actual core dump file with +`coredumpctl dump icinga2 -1 --output `. -systemctl daemon-reload +For further information on how to configure and use coredumpctl, read the man pages +[coredumpctl(1)](https://man7.org/linux/man-pages/man1/coredumpctl.1.html) and +[coredump.conf(5)](https://man7.org/linux/man-pages/man5/coredump.conf.5.html). -systemctl restart icinga2 +#### Ubuntu Apport + +Ubuntu uses their own application `apport` to record core dumps. When it is +enabled, your `core_pattern` will look like this: +``` +# cat /proc/sys/kernel/core_pattern +|/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E ``` -##### Init Script +Apport is unsuitable for development work, because by default it only works +with Ubuntu packages and it has a rather complicated interface for retrieving +the core dump. So unless you rely on Apport for some other workflow, systemd's +coredumpctl is a much better option and is available on Ubuntu in the +`systemd-coredump` package that can replace Apport on your system with no +further setup required. +If you still want to use Apport however, to set it up to work with unpackaged programs, +add the following (create the file if it doesn't exist) to `/etc/apport/settings`: ``` -vim /etc/init.d/icinga2 -... -ulimit -c unlimited - -service icinga2 restart +[main] +unpackaged=true +``` +and restart Apport: +``` +systemctl restart apport.service ``` -##### Verify +When the program crashes you can then find an Apport crash report in `/var/crash/` +that you can read with the interactive `apport-cli` command. To extract the core +dump you run `apport-unpack /var/crash/ ` which then +saves a `/CoreDump` file that contains the actual core dump. -Verify that the Icinga 2 process core file size limit is set to `unlimited`. +#### Directly to a File -``` -for pid in $(pidof icinga2); do cat /proc/$pid/limits; done +If coredumpctl is not available, simply writing the core dump directly to a file +is also sufficient. You can set up your `core_pattern` to write a file to a +suitable path: -... -Max core file size unlimited unlimited bytes +```bash +sysctl -w kernel.core_pattern=/var/lib/cores/core.%e.%p.%h.%t +install -m 1777 -d /var/lib/cores ``` +If you want to make this setting permanent you can also add a file to +`/etc/sysctl.d`, named something like `80-coredumps.conf`: +``` +kernel.core_pattern = /var/lib/cores/core.%e.%p.%h.%t +``` -#### Core Dump Kernel Format +This will create core dump files in `/var/lib/cores` where `%e` is the truncated +name of the program, `%p` is the programs PID, `%h` is the hostname, and `%t` a +timestamp. -The Icinga 2 daemon runs with the SUID bit set. Therefore you need -to explicitly enable core dumps for SUID on Linux. +Note that unlike the other methods this requires the core size limit to be set +for the process. When starting Icinga 2 via systemd you can set it to unlimited +by adding the following to `/etc/systemd/system/icinga2.service.d/limits.conf`: +``` +[Service] +LimitCORE=infinity +``` +Then reload and restart icinga: ```bash -sysctl -w fs.suid_dumpable=2 +systemctl daemon-reload +systemctl restart icinga2.service ``` -Adjust the coredump kernel format and file location on Linux: +Alternatively you edit and reload in one step: +```bash +systemctl edit --drop-in=limits icinga2.service` +``` +When using an init script or starting manually, you need to run `ulimit -c unlimited` +before starting the program: ```bash -sysctl -w kernel.core_pattern=/var/lib/cores/core.%e.%p +ulimit -c unlimited +./icinga2 daemon +``` -install -m 1777 -d /var/lib/cores +To verify that the limit has been set to `unlimited` run the following: +```bash +for pid in $(pidof icinga2); do cat /proc/$pid/limits; done +``` +And look for the line: +``` +Max core file size unlimited unlimited bytes ``` -MacOS: +#### MacOS ```bash sysctl -w kern.corefile=/cores/core.%P - chmod 777 /cores ``` @@ -683,7 +740,7 @@ these tools: - vim - CLion (macOS, Linux) - MS Visual Studio (Windows) -- Atom +- Emacs Editors differ on the functionality. The more helpers you get for C++ development, the faster your development workflow will be. @@ -741,12 +798,12 @@ perfdata | Performance data related, including Graphite, Elastic, etc. db\_ido | IDO database abstraction layer. db\_ido\_mysql | IDO database driver for MySQL. db\_ido\_pgsql | IDO database driver for PgSQL. -mysql\_shin | Library stub for linking against the MySQL client libraries. +mysql\_shim | Library stub for linking against the MySQL client libraries. pgsql\_shim | Library stub for linking against the PgSQL client libraries. #### Class Compiler -Another thing you will recognize are the `.ti` files which are compiled +Something else you might notice are the `.ti` files which are compiled by our own class compiler into actual source code. The meta language allows developers to easily add object attributes and specify their behaviour. @@ -792,17 +849,18 @@ The most common benefits: #### Unity Builds -Another thing you should be aware of: Unity builds on and off. +You should be aware that by default unity builds are enabled. You can turn them +off by setting the `ICINGA2_UNITY_BUILD` CMake option to `OFF`. Typically, we already use caching mechanisms to reduce recompile time with ccache. For release builds, there's always a new build needed as the difference is huge compared to a previous (major) release. -Therefore we've invented the Unity builds, which basically concatenates all source files -into one big library source code file. The compiler then doesn't need to load the many small -files but compiles and links this huge one. +Unity builds basically concatenate all source files into one big library source code file. +The compiler then doesn't need to load many small files, each with all of their includes, +but compiles and links only a few huge ones. -Unity builds require more memory which is why you should disable them for development +However, unity builds require more memory which is why you should disable them for development builds in small sized VMs (Linux, Windows) and also Docker containers. There's a couple of header files which are included everywhere. If you touch/edit them, @@ -1228,7 +1286,7 @@ every second. Avoid log messages which could irritate the user. During implementation, developers can change log levels to better -see what's going one, but remember to change this back to `debug` +see what's going on, but remember to change this back to `debug` or remove it entirely. @@ -2262,7 +2320,7 @@ cmake .. -DCMAKE_INSTALL_PREFIX=/tmp/icinga2 ### CMake Variables -In addition to `CMAKE_INSTALL_PREFIX` here are most of the supported Icinga-specific cmake variables. +In addition to `CMAKE_INSTALL_PREFIX` here are most of the supported Icinga-specific CMake variables. For all variables regarding defaults paths on in CMake, see [GNUInstallDirs](https://cmake.org/cmake/help/latest/module/GNUInstallDirs.html).