HA: Show SQL Process List On Failure

[oxzi]

In case the HA realization logic is aborted for exceeding its context deadline, a nicer user output would be desirable. One idea would be to try to acquire the SQL process list and also present it to the user, indicating why the database is overworked.

Currently, it may look like the following.

FATAL  icingadb  pq: canceling statement due to user request
can't perform "INSERT INTO \"icingadb_instance\" (\"id\", \"heartbeat\", \"icinga2_version\", \"icinga2_flap_detection_enabled\", \"icinga2_performance_data_enabled\", \"endpoint_id\", \"icinga2_notifications_enabled\", \"icinga2_active_host_checks_enabled\", \"icinga2_event_handlers_enabled\", \"environment_id\", \"responsible\", \"icinga2_start_time\", \"icinga2_active_service_checks_enabled\") VALUES (:id,:heartbeat,:icinga2_version,:icinga2_flap_detection_enabled,:icinga2_performance_data_enabled,:endpoint_id,:icinga2_notifications_enabled,:icinga2_active_host_checks_enabled,:icinga2_event_handlers_enabled,:environment_id,:responsible,:icinga2_start_time,:icinga2_active_service_checks_enabled) ON CONFLICT ON CONSTRAINT pk_icingadb_instance DO UPDATE SET \"id\" = EXCLUDED.\"id\",\"heartbeat\" = EXCLUDED.\"heartbeat\",\"icinga2_version\" = EXCLUDED.\"icinga2_version\",\"icinga2_flap_detection_enabled\" = EXCLUDED.\"icinga2_flap_detection_enabled\",\"icinga2_performance_data_enabled\" = EXCLUDED.\"icinga2_performance_data_enabled\",\"endpoint_id\" = EXCLUDED.\"endpoint_id\",\"icinga2_notifications_enabled\" = EXCLUDED.\"icinga2_notifications_enabled\",\"icinga2_active_host_checks_enabled\" = EXCLUDED.\"icinga2_active_host_checks_enabled\",\"icinga2_event_handlers_enabled\" = EXCLUDED.\"icinga2_event_handlers_enabled\",\"environment_id\" = EXCLUDED.\"environment_id\",\"responsible\" = EXCLUDED.\"responsible\",\"icinga2_start_time\" = EXCLUDED.\"icinga2_start_time\",\"icinga2_active_service_checks_enabled\" = EXCLUDED.\"icinga2_active_service_checks_enabled\""
github.com/icinga/icinga-go-library/database.CantPerformQuery
  github.com/icinga/icinga-go-library@v0.6.3/database/utils.go:16
github.com/icinga/icingadb/pkg/icingadb.(*HA).realize.func1
  github.com/icinga/icingadb/pkg/icingadb/ha.go:383
github.com/icinga/icinga-go-library/retry.WithBackoff
  github.com/icinga/icinga-go-library@v0.6.3/retry/retry.go:65
github.com/icinga/icingadb/pkg/icingadb.(*HA).realize
  github.com/icinga/icingadb/pkg/icingadb/ha.go:292
github.com/icinga/icingadb/pkg/icingadb.(*HA).controller
  github.com/icinga/icingadb/pkg/icingadb/ha.go:224
runtime.goexit
  runtime/asm_amd64.s:1700
HA aborted
github.com/icinga/icingadb/pkg/icingadb.(*HA).abort.func1
  github.com/icinga/icingadb/pkg/icingadb/ha.go:134
sync.(*Once).doSlow
  sync/once.go:78
sync.(*Once).Do
  sync/once.go:69
github.com/icinga/icingadb/pkg/icingadb.(*HA).abort
  github.com/icinga/icingadb/pkg/icingadb/ha.go:132
github.com/icinga/icingadb/pkg/icingadb.(*HA).controller
  github.com/icinga/icingadb/pkg/icingadb/ha.go:236
runtime.goexit
  runtime/asm_amd64.s:1700
HA exited with an error
main.run
  github.com/icinga/icingadb/cmd/icingadb/main.go:347
main.main
  github.com/icinga/icingadb/cmd/icingadb/main.go:37
runtime.main
  runtime/proc.go:283
runtime.goexit
  runtime/asm_amd64.s:1700

The other issue #933 is semi-related.

[Al2Klimov]

Wouldn't this be a nice feature for our Go-DB stuff in general? (igl)