Service notified before ScheduledDowntime was activated

[log1-c]

Describe the bug

We are currently facing a problem, where during a deployment of hosts, those hosts or services of them send out notifications despite them supposedly being in a downtime

The hosts get imported from our Netbox and all have a tag "recently_deployed". As long as this tag is present, a ScheduledDowntime is deployed to the host and all its services.

This mostly works really well, but there are occasions where one service still sends a notification, because it gets triggered before the downtime is activated on the service.

Example from today:

Config Deployment started via Director at 13:46 (for reference icinga2 daemon -C takes about 1m55s and we have a setup with ~110 satellites/zones)

This is the ScheduledDowntime config (configured inside the Director)

zones.d/master/scheduled_downtime_apply.conf
apply ScheduledDowntime "VM creation downtime" to Host {
    author = "netbox-cloud-importer"
    comment = "This downtime will be deployed as long as the host has the 'recently_deployed' tag set on its netbox VM object"
    fixed = true
    assign where host.vars.tags.recently_deployed == "true"
    ranges = {
        "friday"	= "00:00-24:00"
        "monday"	= "00:00-24:00"
        "saturday"	= "00:00-24:00"
        "sunday"	= "00:00-24:00"
        "thursday"	= "00:00-24:00"
        "tuesday"	= "00:00-24:00"
        "wednesday"	= "00:00-24:00"
    }
}

apply ScheduledDowntime "VM creation downtime" to Service {
    author = "netbox-cloud-importer"
    comment = "This downtime will be deployed as long as the host has the 'recently_deployed' tag set on its netbox VM object"
    fixed = true
    assign where host.vars.tags.recently_deployed == "true"
    ranges = {
        "friday"	= "00:00-24:00"
        "monday"	= "00:00-24:00"
        "saturday"	= "00:00-24:00"
        "sunday"	= "00:00-24:00"
        "thursday"	= "00:00-24:00"
        "tuesday"	= "00:00-24:00"
        "wednesday"	= "00:00-24:00"
    }
}

What we got from the Debug log concerning one specific host

#Check soft-state change to CRITICAL 
[2025-08-28 13:48:18 +0200] notice/Checkable: State Change: Checkable 'abcd-p1-ts1130!icinga-agent-parent-service' soft state change from UNKNOWN to CRITICAL detected.

#Host state changed to UP
[2025-08-28 13:48:23 +0200] notice/Checkable: State Change: Checkable 'abcd-p1-ts1130' hard state change from DOWN to UP detected.

#Check hard-state change to CRITICAL and notification sent
[2025-08-28 13:50:12 +0200] notice/Checkable: State Change: Checkable 'abcd-p1-ts1130!icinga-agent-parent-service' hard state change from CRITICAL to CRITICAL detected.
[2025-08-28 13:50:12 +0200] information/Checkable: Checkable 'abcd-p1-ts1130!icinga-agent-parent-service' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2025-08-28 13:50:12 +0200] notice/Notification: Attempting to send notifications of type 'Problem' for notification object 'abcd-p1-ts1130!icinga-agent-parent-service!notification-jira-service_critical'.
[2025-08-28 13:50:12 +0200] debug/Notification: User 'abcd-ticketsystem' notification 'abcd-p1-ts1130!icinga-agent-parent-service!notification-jira-service_critical', Type 'Problem', TypeFilter: Custom, Problem and Recovery (FType=32, TypeFilter=104)
[2025-08-28 13:50:12 +0200] debug/Notification: User 'abcd-ticketsystem' notification 'abcd-p1-ts1130!icinga-agent-parent-service!notification-jira-service_critical', State 'Critical', StateFilter: Critical, Down, OK and Up (FState=4, StateFilter=53)
[2025-08-28 13:50:12 +0200] information/Notification: Sending 'Problem' notification 'abcd-p1-ts1130!icinga-agent-parent-service!notification-jira-service_critical' for user 'abcd-ticketsystem'
[2025-08-28 13:50:12 +0200] notice/ClusterEvents: Discarding 'send custom notification' message for checkable 'abcd-p1-ts1130!icinga-agent-parent-service' from 'abcd-lm1-mde01': Unauthorized access.
[2025-08-28 13:50:12 +0200] notice/Process: Running command '/etc/icinga2/scripts/icinga-jira-notification-scripts/icingacli_webhook/icingacli-webhook.py' 'jira' 'send' 'problem' '--auswirkung' '10001' '--description' '' '--dringlichkeit' '10004' '--fallback-email' 'jira-@abcd.cloud' '--host' 'abcd-p1-ts1130' '--issuetype' 'Incident' '--kundenkuerzel' 'abcd' '--project' 'abcd' '--rbgroup' 'RB Group 2' '--service' 'icinga-agent-parent-service' '--serviceowner' 'Virtual Desktop' '--state' 'CRITICAL' '--summary' 'icinga-agent-parent-service on abcd-p1-ts1130 is CRITICAL at 2025-08-28 13:48:18 +0200' '--template' 'abcd-ticketsystem' '--ticketsystem' 'abcd': PID 2127088
[2025-08-28 13:50:12 +0200] information/Notification: Completed sending 'Problem' notification 'abcd-p1-ts1130!icinga-agent-parent-service!notification-jira-service_critical' for checkable 'abcd-p1-ts1130!icinga-agent-parent-service' and user 'abcd-ticketsystem' using command 'JIRA Service Notification'.

#Acknowledgement from JIRA module for the notified check
[2025-08-28 13:50:14 +0200] notice/ConfigCompiler: Compiling config file: /var/lib/icinga2/api/packages/_api/8de75f7d-91bc-4599-8f75-51a4fb95586d/conf.d/comments/abcd-p1-ts1130!icinga-agent-parent-service!5033cafb-04c0-40e8-bd70-6ea6a8bf0eff.conf
[2025-08-28 13:50:14 +0200] information/ConfigObjectUtility: Created and activated object 'abcd-p1-ts1130!icinga-agent-parent-service!5033cafb-04c0-40e8-bd70-6ea6a8bf0eff' of type 'Comment'.
[2025-08-28 13:50:14 +0200] notice/Comment: Added comment 'abcd-p1-ts1130!icinga-agent-parent-service!5033cafb-04c0-40e8-bd70-6ea6a8bf0eff'.
[2025-08-28 13:50:14 +0200] information/Checkable: Acknowledgement set for checkable 'abcd-p1-ts1130!icinga-agent-parent-service'.
[2025-08-28 13:50:14 +0200] notice/Process: PID 2127088 ('/etc/icinga2/scripts/icinga-jira-notification-scripts/icingacli_webhook/icingacli-webhook.py' 'jira' 'send' 'problem' '--auswirkung' '10001' '--description' '' '--dringlichkeit' '10004' '--fallback-email' 'jira-@abcd.cloud' '--host' 'abcd-p1-ts1130' '--issuetype' 'Incident' '--kundenkuerzel' 'abcd' '--project' 'abcd' '--rbgroup' 'RB Group 2' '--service' 'icinga-agent-parent-service' '--serviceowner' 'Virtual Desktop' '--state' 'CRITICAL' '--summary' 'icinga-agent-parent-service on abcd-p1-ts1130 is CRITICAL at 2025-08-28 13:48:18 +0200' '--template' 'abcd-ticketsystem' '--ticketsystem' 'abcd') terminated with exit code 0
[2025-08-28 13:50:14 +0200] notice/NotificationComponent: Reminder notification 'abcd-p1-ts1130!icinga-agent-parent-service!notification-jira-service_critical': Notification was sent out once and interval=0 disables reminder notifications.

#ScheduledDowntime for current day gets created for service
[2025-08-28 13:50:19 +0200] debug/ScheduledDowntime: Creating new Downtime for ScheduledDowntime "abcd-p1-ts1130!icinga-agent-parent-service!VM creation downtime"
[2025-08-28 13:50:19 +0200] notice/ConfigCompiler: Compiling config file: /var/lib/icinga2/api/packages/_api/8de75f7d-91bc-4599-8f75-51a4fb95586d/conf.d/downtimes/abcd-p1-ts1130!icinga-agent-parent-service!6bb9f7b2-f47e-49eb-9b3f-07e2f8477772.conf
[2025-08-28 13:50:19 +0200] information/Checkable: Checkable 'abcd-p1-ts1130!icinga-agent-parent-service' has 1 notification(s). Checking filters for type 'DowntimeStart', sends will be logged.
[2025-08-28 13:50:19 +0200] notice/Notification: Attempting to send notifications of type 'DowntimeStart' for notification object 'abcd-p1-ts1130!icinga-agent-parent-service!notification-jira-service_critical'.
[2025-08-28 13:50:19 +0200] notice/Notification: Not sending notifications for notification object 'abcd-p1-ts1130!icinga-agent-parent-service!notification-jira-service_critical': type 'DowntimeStart' does not match type filter: Custom, Problem and Recovery.
[2025-08-28 13:50:19 +0200] information/Downtime: Triggering downtime 'abcd-p1-ts1130!icinga-agent-parent-service!6bb9f7b2-f47e-49eb-9b3f-07e2f8477772' for checkable 'abcd-p1-ts1130!icinga-agent-parent-service'.
[2025-08-28 13:50:19 +0200] information/ConfigObjectUtility: Created and activated object 'abcd-p1-ts1130!icinga-agent-parent-service!6bb9f7b2-f47e-49eb-9b3f-07e2f8477772' of type 'Downtime'.
[2025-08-28 13:50:19 +0200] information/Downtime: Added downtime 'abcd-p1-ts1130!icinga-agent-parent-service!6bb9f7b2-f47e-49eb-9b3f-07e2f8477772' between '2025-08-28 00:00:00' and '2025-08-29 00:00:00', author: 'netbox-cloud-importer', fixed
[2025-08-28 13:50:19 +0200] notice/ClusterEvents: Discarding 'send custom notification' message for checkable 'abcd-p1-ts1130!icinga-agent-parent-service' from 'abcd-lm1-mde01': Unauthorized access.

#ScheduledDowntime for current day gets created for host
[2025-08-28 13:50:20 +0200] notice/NotificationComponent: Reminder notification 'abcd-p1-ts1130!icinga-agent-parent-service!notification-jira-service_critical': Notification was sent out once and interval=0 disables reminder notifications.
[2025-08-28 13:50:25 +0200] notice/NotificationComponent: Reminder notification 'abcd-p1-ts1130!icinga-agent-parent-service!notification-jira-service_critical': Notification was sent out once and interval=0 disables reminder notifications.
[2025-08-28 13:50:28 +0200] debug/ScheduledDowntime: Creating new Downtime for ScheduledDowntime "abcd-p1-ts1130!VM creation downtime"
[2025-08-28 13:50:28 +0200] notice/ConfigCompiler: Compiling config file: /var/lib/icinga2/api/packages/_api/8de75f7d-91bc-4599-8f75-51a4fb95586d/conf.d/downtimes/abcd-p1-ts1130!42941894-f120-4ccd-a47f-bffbb4767810.conf
[2025-08-28 13:50:28 +0200] information/Checkable: Checkable 'abcd-p1-ts1130' has 1 notification(s). Checking filters for type 'DowntimeStart', sends will be logged.
[2025-08-28 13:50:28 +0200] notice/Notification: Attempting to send notifications of type 'DowntimeStart' for notification object 'abcd-p1-ts1130!notification-jira-host_down'.
[2025-08-28 13:50:28 +0200] notice/Notification: Not sending notifications for notification object 'abcd-p1-ts1130!notification-jira-host_down': type 'DowntimeStart' does not match type filter: Custom, Problem and Recovery.
[2025-08-28 13:50:28 +0200] information/Downtime: Triggering downtime 'abcd-p1-ts1130!42941894-f120-4ccd-a47f-bffbb4767810' for checkable 'abcd-p1-ts1130'.
[2025-08-28 13:50:28 +0200] information/ConfigObjectUtility: Created and activated object 'abcd-p1-ts1130!42941894-f120-4ccd-a47f-bffbb4767810' of type 'Downtime'.
[2025-08-28 13:50:28 +0200] information/Downtime: Added downtime 'abcd-p1-ts1130!42941894-f120-4ccd-a47f-bffbb4767810' between '2025-08-28 00:00:00' and '2025-08-29 00:00:00', author: 'netbox-cloud-importer', fixed
[2025-08-28 13:50:28 +0200] notice/ClusterEvents: Discarding 'send custom notification' message for checkable 'abcd-p1-ts1130' from 'abcd-lm1-mde01': Unauthorized access.
... many other services of that host

#ScheduledDowntime for next day gets created for service
[2025-08-28 13:52:54 +0200] debug/ScheduledDowntime: Creating new Downtime for ScheduledDowntime "abcd-p1-ts1130!icinga-agent-parent-service!VM creation downtime"
[2025-08-28 13:52:54 +0200] notice/ConfigCompiler: Compiling config file: /var/lib/icinga2/api/packages/_api/8de75f7d-91bc-4599-8f75-51a4fb95586d/conf.d/downtimes/abcd-p1-ts1130!icinga-agent-parent-service!7580a912-fbca-4a26-9cc5-ec590c87222e.conf
[2025-08-28 13:52:54 +0200] information/ConfigObjectUtility: Created and activated object 'abcd-p1-ts1130!icinga-agent-parent-service!7580a912-fbca-4a26-9cc5-ec590c87222e' of type 'Downtime'.
[2025-08-28 13:52:54 +0200] information/Downtime: Added downtime 'abcd-p1-ts1130!icinga-agent-parent-service!7580a912-fbca-4a26-9cc5-ec590c87222e' between '2025-08-29 00:00:00' and '2025-08-30 00:00:00', author: 'netbox-cloud-importer', fixed
[2025-08-28 13:52:55 +0200] notice/NotificationComponent: Reminder notification 'abcd-p1-ts1130!icinga-agent-parent-service!notification-jira-service_critical': Notification was sent out once and interval=0 disables reminder notifications.
[2025-08-28 13:53:00 +0200] notice/NotificationComponent: Reminder notification 'abcd-p1-ts1130!icinga-agent-parent-service!notification-jira-service_critical': Notification was sent out once and interval=0 disables reminder notifications.

#ScheduledDowntime for next day gets created for host
[2025-08-28 13:53:03 +0200] debug/ScheduledDowntime: Creating new Downtime for ScheduledDowntime "abcd-p1-ts1130!VM creation downtime"
[2025-08-28 13:53:03 +0200] notice/ConfigCompiler: Compiling config file: /var/lib/icinga2/api/packages/_api/8de75f7d-91bc-4599-8f75-51a4fb95586d/conf.d/downtimes/abcd-p1-ts1130!fe0d1fdf-ebf1-42ed-a5f4-514b426dcf1b.conf
[2025-08-28 13:53:03 +0200] information/ConfigObjectUtility: Created and activated object 'abcd-p1-ts1130!fe0d1fdf-ebf1-42ed-a5f4-514b426dcf1b' of type 'Downtime'.
[2025-08-28 13:53:03 +0200] information/Downtime: Added downtime 'abcd-p1-ts1130!fe0d1fdf-ebf1-42ed-a5f4-514b426dcf1b' between '2025-08-29 00:00:00' and '2025-08-30 00:00:00', author: 'netbox-cloud-importer', fixed
[2025-08-28 13:53:05 +0200] notice/NotificationComponent: Reminder notification 'abcd-p1-ts1130!icinga-agent-parent-service!notification-jira-service_critical': Notification was sent out once and interval=0 disables reminder notifications.

To Reproduce

not sure?

Expected behavior

Hosts/services that are (should) be in a downtime, should not send notifications.

Your Environment

Include as many relevant details about the environment you experienced the problem in

• Version used (icinga2 --version): 2.15.0
• Operating System and version: rhel8
• Enabled features (icinga2 feature list): api checker debuglog icingadb influxdb2 mainlog notification syslog
• Icinga Web 2 version and modules (System - About): 2.12.5
• Config validation (icinga2 daemon -C):

[2025-08-28 13:46:57 +0200] information/ConfigItem: Instantiated 1 NotificationComponent.
[2025-08-28 13:46:57 +0200] information/ConfigItem: Instantiated 1 SyslogLogger.
[2025-08-28 13:46:57 +0200] information/ConfigItem: Instantiated 146 HostGroups.
[2025-08-28 13:46:57 +0200] information/ConfigItem: Instantiated 1 CheckerComponent.
[2025-08-28 13:46:57 +0200] information/ConfigItem: Instantiated 18194 Dependencies.
[2025-08-28 13:46:57 +0200] information/ConfigItem: Instantiated 39 Users.
[2025-08-28 13:46:57 +0200] information/ConfigItem: Instantiated 9 UserGroups.
[2025-08-28 13:46:57 +0200] information/ConfigItem: Instantiated 4 TimePeriods.
[2025-08-28 13:46:57 +0200] information/ConfigItem: Instantiated 4 ServiceGroups.
[2025-08-28 13:46:57 +0200] information/ConfigItem: Instantiated 53173 Services.
[2025-08-28 13:46:57 +0200] information/ConfigItem: Instantiated 61068 ScheduledDowntimes.
[2025-08-28 13:46:57 +0200] information/ConfigItem: Instantiated 1267 Zones.
[2025-08-28 13:46:57 +0200] information/ConfigItem: Instantiated 47673 Notifications.
[2025-08-28 13:46:57 +0200] information/ConfigItem: Instantiated 6 NotificationCommands.
[2025-08-28 13:46:57 +0200] information/ConfigItem: Instantiated 2 FileLoggers.
[2025-08-28 13:46:57 +0200] information/ConfigItem: Instantiated 1 IcingaApplication.
[2025-08-28 13:46:57 +0200] information/ConfigItem: Instantiated 2363 Hosts.
[2025-08-28 13:46:57 +0200] information/ConfigItem: Instantiated 8 EventCommands.
[2025-08-28 13:46:57 +0200] information/ConfigItem: Instantiated 16446 Downtimes.
[2025-08-28 13:46:57 +0200] information/ConfigItem: Instantiated 1265 Endpoints.
[2025-08-28 13:46:57 +0200] information/ConfigItem: Instantiated 60 Comments.
[2025-08-28 13:46:57 +0200] information/ConfigItem: Instantiated 92 ApiUsers.
[2025-08-28 13:46:57 +0200] information/ConfigItem: Instantiated 1 ApiListener.
[2025-08-28 13:46:57 +0200] information/ConfigItem: Instantiated 508 CheckCommands.
[2025-08-28 13:46:57 +0200] information/ConfigItem: Instantiated 1 IcingaDB.
[2025-08-28 13:46:57 +0200] information/ConfigItem: Instantiated 3 Influxdb2Writers.

Any help/insights appreciated.
In case you need further information, please ask.

Cheers!

[julianbrost]

ScheduledDowntimes basically only regularly create normal Downtime objects (just as your logs show), which happens in background. Typically, these are created early enough, but if everything is new, the checker might just be faster checking the service for the first time than the ScheduledDowntime was for creating the first Downtime for it.

Code reference: ScheduledDowntime::Start():

icinga2/lib/icinga/scheduleddowntime.cpp

Line 88 in 0c2fd00

Utility::QueueAsyncCallback([this]() { CreateNextDowntime(); });

Open question: would synchronously creating the Downtime objects create any problems there (like activating config creates more config)?

Possible workarounds: use higher max_check_attempts or set times.begin in the Notification objects if that recently_deployed is set, both with the goal of delaying the notification a bit more so that there's a higher chance that the Downtime was created in time.