Very Long Delay Behind MS App Proxy before Data Shown

[austinjhunt]

We have two Icinga environments, one for test and one for prod. In production, we have our Master (icingaweb2) server behind a proxy, this is the URL: https://icinga-cofc.msappproxy.net

We have the dashing dashboard located at https://icinga-cofc.msappproxy.net/dashingicinga

When you access dashing via the server name directly (say, http://icingawebserver/dashingicinga), it works like a charm.

However, when you access via the proxy URL, it also works, but data isn't loaded into the widgets until after a significant delay (1m55s according to my stopwatch).

More interestingly, running dashing in the foreground seems to indicate that data is being retrieved at the SCHEDULER interval of 10 seconds, but the front end is not updating to reflect those retrievals. At least, again, not until after a long time.

Wondering what sort of variables/intervals in the source do you think align with this time frame and is it fixable? Have you seen a working instance of dashing-icinga2 behind a proxy?

This is a captured log of running dashing in the foreground while trying to access dashing via proxy URL

[root@RP-Icinga-M01 dashing-icinga2]# head -n 100 dashing-issue-trace.log
First trying to read environment variables
Environment variables not found, falling back to configuration file config/icinga2.local.json
Detecting local config file '/usr/share/dashing-icinga2/config/icinga2.local.local.json'.
Using config file '/usr/share/dashing-icinga2/config/icinga2.local.json'.
Reading config{"icinga2"=>{"api"=>{"host"=>"localhost", "port"=>5665, "user"=>"dashing", "password"=>"not including"}}, "icingaweb2"=>{"url"=>"/"}, "dashboard"=>{"show_only_hard_state_problems"=>false, "timezone"=>"America/New_York"}}
Thin web server (v1.6.4 codename Gob Bluth)
Maximum connections set to 1024
Listening on 0.0.0.0:8005, CTRL+C to stop
App Info: {"icingaapplication"=>{"app"=>{"enable_event_handlers"=>true, "enable_flapping"=>true, "enable_host_checks"=>true, "enable_notifications"=>true, "enable_perfdata"=>true, "enable_service_checks"=>true, "environment"=>"", "node_name"=>"IcingaMaster", "pid"=>7415.0, "program_start"=>1585774143.078506, "version"=>"2.11.2-1"}}} Version: 2.11.2
Meter widget: Hosts 81.0/1131 Services 122.0/2196
conn_endpoints ["IcingaMaster", "IcingaMaster", "IcingaMaster"]
http {"clients"=>1.0}
identity IcingaMaster
json_rpc {"anonymous_clients"=>0.0, "relay_queue_item_rate"=>245.8, "relay_queue_items"=>0.0, "sync_queue_item_rate"=>0.0, "sync_queue_items"=>0.0, "work_queue_item_rate"=>184.41666666666666}
not_conn_endpoints []
num_conn_endpoints 3.0
num_endpoints 3.0
num_not_conn_endpoints 0.0
zones {"IZ-A"=>{"client_log_lag"=>0.0, "connected"=>true, "endpoints"=>["IcingaMaster", "IcingaMaster"], "parent_zone"=>"IZ-MASTER"}, "IZ-B"=>{"client_log_lag"=>0.0, "connected"=>true, "endpoints"=>["IcingaMaster"], "parent_zone"=>"IZ-MASTER"}, "IZ-MASTER"=>{"client_log_lag"=>0.0, "connected"=>true, "endpoints"=>["IcingaMaster"], "parent_zone"=>""}}
checker {"idle"=>32.0, "pending"=>0.0}
command 1.0
main-log 1.0
app {"enable_event_handlers"=>true, "enable_flapping"=>true, "enable_host_checks"=>true, "enable_notifications"=>true, "enable_perfdata"=>true, "enable_service_checks"=>true, "environment"=>"", "node_name"=>"IcingaMaster", "pid"=>7415.0, "program_start"=>1585774143.078506, "version"=>"2.11.2-1"}
ido-mysql {"connected"=>true, "instance_name"=>"default", "query_queue_item_rate"=>125.46666666666667, "query_queue_items"=>0.0, "version"=>"1.14.3"}
influxdb {"data_buffer_items"=>390.0, "work_queue_item_rate"=>61.483333333333334, "work_queue_items"=>0.0}
notification 1.0
Stats: [{"label"=>"Host checks/min", "value"=>264.0}, {"label"=>"Service checks/min", "value"=>3459.0}, {"label"=>"json_rpc queue rate", "value"=>"184.42"}, {"label"=>"ido-mysql queue rate", "value"=>"125.47"}, {"label"=>"influxdb queue rate", "value"=>"61.48"}]
Severity: [{"label"=>"360CON-3FE-R3-1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"JCLDCSRX - hostalive", "color"=>"red", "state"=>2}, {"label"=>"170CAL-PCI-PA500HA-1F1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"BELL529-PA7050HA-5F2 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"360CON-3FW-R3-1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"ARENA-2FW-R1-1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"24GEO-3FN-R2-2 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"ARENA-2FE-R0-2 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"360CON-1FW1-R1-2 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"133CAL-2F1--RE-N0 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"80BSTP-2FE-R1-1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"360CON-3FW-R2-2 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"BELL-3F-R1-1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"360CON-3FE-R1-1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"BELL418-PA850-VPN-4F1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"74GEO-1F-R1-1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"PPMAIN-3F-R1-1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"RHSC-3FW-R2-1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"RSS-1F-R2-1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"85PAT-BAT-1F1-R1-1 - hostalive", "color"=>"red", "state"=>2}]
First trying to read environment variables
Environment variables not found, falling back to configuration file config/icinga2.json
Detecting local config file '/usr/share/dashing-icinga2/config/icinga2.local.json'.
Using config file '/usr/share/dashing-icinga2/config/icinga2.local.json'.
Reading config{"icinga2"=>{"api"=>{"host"=>"localhost", "port"=>5665, "user"=>"dashing", "password"=>"not including"}}, "icingaweb2"=>{"url"=>"/"}, "dashboard"=>{"show_only_hard_state_problems"=>false, "timezone"=>"America/New_York"}}
First trying to read environment variables
Environment variables not found, falling back to configuration file config/icinga2.json
Detecting local config file '/usr/share/dashing-icinga2/config/icinga2.local.json'.
Using config file '/usr/share/dashing-icinga2/config/icinga2.local.json'.
Reading config{"icinga2"=>{"api"=>{"host"=>"localhost", "port"=>5665, "user"=>"dashing", "password"=>"not including"}}, "icingaweb2"=>{"url"=>"/"}, "dashboard"=>{"show_only_hard_state_problems"=>false, "timezone"=>"America/New_York"}}
App Info: {"icingaapplication"=>{"app"=>{"enable_event_handlers"=>true, "enable_flapping"=>true, "enable_host_checks"=>true, "enable_notifications"=>true, "enable_perfdata"=>true, "enable_service_checks"=>true, "environment"=>"", "node_name"=>"IcingaMaster", "pid"=>7415.0, "program_start"=>1585774143.078506, "version"=>"2.11.2-1"}}} Version: 2.11.2
Meter widget: Hosts 81.0/1131 Services 122.0/2196
conn_endpoints ["IcingaMaster", "IcingaMaster", "IcingaMaster"]
http {"clients"=>1.0}
identity IcingaMaster
json_rpc {"anonymous_clients"=>0.0, "relay_queue_item_rate"=>244.8, "relay_queue_items"=>0.0, "sync_queue_item_rate"=>0.0, "sync_queue_items"=>0.0, "work_queue_item_rate"=>183.71666666666667}
not_conn_endpoints []
num_conn_endpoints 3.0
num_endpoints 3.0
num_not_conn_endpoints 0.0
zones {"IZ-A"=>{"client_log_lag"=>0.0, "connected"=>true, "endpoints"=>["IcingaMaster", "IcingaMaster"], "parent_zone"=>"IZ-MASTER"}, "IZ-B"=>{"client_log_lag"=>0.0, "connected"=>true, "endpoints"=>["IcingaMaster"], "parent_zone"=>"IZ-MASTER"}, "IZ-MASTER"=>{"client_log_lag"=>0.0, "connected"=>true, "endpoints"=>["IcingaMaster"], "parent_zone"=>""}}
checker {"idle"=>32.0, "pending"=>0.0}
command 1.0
main-log 1.0
app {"enable_event_handlers"=>true, "enable_flapping"=>true, "enable_host_checks"=>true, "enable_notifications"=>true, "enable_perfdata"=>true, "enable_service_checks"=>true, "environment"=>"", "node_name"=>"IcingaMaster", "pid"=>7415.0, "program_start"=>1585774143.078506, "version"=>"2.11.2-1"}
ido-mysql {"connected"=>true, "instance_name"=>"default", "query_queue_item_rate"=>124.93333333333334, "query_queue_items"=>0.0, "version"=>"1.14.3"}
influxdb {"data_buffer_items"=>497.0, "work_queue_item_rate"=>61.233333333333334, "work_queue_items"=>0.0}
notification 1.0
Stats: [{"label"=>"Host checks/min", "value"=>270.0}, {"label"=>"Service checks/min", "value"=>3426.0}, {"label"=>"json_rpc queue rate", "value"=>"183.72"}, {"label"=>"ido-mysql queue rate", "value"=>"124.93"}, {"label"=>"influxdb queue rate", "value"=>"61.23"}]
Severity: [{"label"=>"360CON-3FE-R3-1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"JCLDCSRX - hostalive", "color"=>"red", "state"=>2}, {"label"=>"170CAL-PCI-PA500HA-1F1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"BELL529-PA7050HA-5F2 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"360CON-3FW-R3-1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"ARENA-2FW-R1-1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"24GEO-3FN-R2-2 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"ARENA-2FE-R0-2 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"360CON-1FW1-R1-2 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"133CAL-2F1--RE-N0 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"80BSTP-2FE-R1-1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"360CON-3FW-R2-2 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"BELL-3F-R1-1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"360CON-3FE-R1-1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"BELL418-PA850-VPN-4F1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"74GEO-1F-R1-1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"PPMAIN-3F-R1-1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"RHSC-3FW-R2-1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"RSS-1F-R2-1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"85PAT-BAT-1F1-R1-1 - hostalive", "color"=>"red", "state"=>2}]
App Info: {"icingaapplication"=>{"app"=>{"enable_event_handlers"=>true, "enable_flapping"=>true, "enable_host_checks"=>true, "enable_notifications"=>true, "enable_perfdata"=>true, "enable_service_checks"=>true, "environment"=>"", "node_name"=>"IcingaMaster", "pid"=>7415.0, "program_start"=>1585774143.078506, "version"=>"2.11.2-1"}}} Version: 2.11.2
Meter widget: Hosts 81.0/1131 Services 122.0/2196
conn_endpoints ["IcingaMaster", "IcingaMaster", "IcingaMaster"]
http {"clients"=>1.0}
identity IcingaMaster
json_rpc {"anonymous_clients"=>0.0, "relay_queue_item_rate"=>247.28333333333333, "relay_queue_items"=>0.0, "sync_queue_item_rate"=>0.0, "sync_queue_items"=>0.0, "work_queue_item_rate"=>185.46666666666667}
not_conn_endpoints []
num_conn_endpoints 3.0
num_endpoints 3.0
num_not_conn_endpoints 0.0
zones {"IZ-A"=>{"client_log_lag"=>0.0, "connected"=>true, "endpoints"=>["IcingaMaster", "IcingaMaster"], "parent_zone"=>"IZ-MASTER"}, "IZ-B"=>{"client_log_lag"=>0.0, "connected"=>true, "endpoints"=>["IcingaMaster"], "parent_zone"=>"IZ-MASTER"}, "IZ-MASTER"=>{"client_log_lag"=>0.0, "connected"=>true, "endpoints"=>["IcingaMaster"], "parent_zone"=>""}}
checker {"idle"=>32.0, "pending"=>0.0}
command 1.0
main-log 1.0
app {"enable_event_handlers"=>true, "enable_flapping"=>true, "enable_host_checks"=>true, "enable_notifications"=>true, "enable_perfdata"=>true, "enable_service_checks"=>true, "environment"=>"", "node_name"=>"IcingaMaster", "pid"=>7415.0, "program_start"=>1585774143.078506, "version"=>"2.11.2-1"}
ido-mysql {"connected"=>true, "instance_name"=>"default", "query_queue_item_rate"=>126.36666666666666, "query_queue_items"=>0.0, "version"=>"1.14.3"}
influxdb {"data_buffer_items"=>400.0, "work_queue_item_rate"=>61.93333333333333, "work_queue_items"=>0.0}
notification 1.0
Stats: [{"label"=>"Host checks/min", "value"=>266.0}, {"label"=>"Service checks/min", "value"=>3409.0}, {"label"=>"json_rpc queue rate", "value"=>"185.47"}, {"label"=>"ido-mysql queue rate", "value"=>"126.37"}, {"label"=>"influxdb queue rate", "value"=>"61.93"}]
Severity: [{"label"=>"360CON-3FE-R3-1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"JCLDCSRX - hostalive", "color"=>"red", "state"=>2}, {"label"=>"170CAL-PCI-PA500HA-1F1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"BELL529-PA7050HA-5F2 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"360CON-3FW-R3-1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"ARENA-2FW-R1-1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"24GEO-3FN-R2-2 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"ARENA-2FE-R0-2 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"360CON-1FW1-R1-2 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"133CAL-2F1--RE-N0 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"80BSTP-2FE-R1-1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"360CON-3FW-R2-2 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"BELL-3F-R1-1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"360CON-3FE-R1-1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"BELL418-PA850-VPN-4F1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"74GEO-1F-R1-1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"PPMAIN-3F-R1-1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"RHSC-3FW-R2-1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"RSS-1F-R2-1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"85PAT-BAT-1F1-R1-1 - hostalive", "color"=>"red", "state"=>2}]
App Info: {"icingaapplication"=>{"app"=>{"enable_event_handlers"=>true, "enable_flapping"=>true, "enable_host_checks"=>true, "enable_notifications"=>true, "enable_perfdata"=>true, "enable_service_checks"=>true, "environment"=>"", "node_name"=>"IcingaMaster", "pid"=>7415.0, "program_start"=>1585774143.078506, "version"=>"2.11.2-1"}}} Version: 2.11.2
Meter widget: Hosts 81.0/1131 Services 121.0/2196
conn_endpoints ["IcingaMaster", "IcingaMaster", "IcingaMaster"]
http {"clients"=>1.0}
identity IcingaMaster
json_rpc {"anonymous_clients"=>0.0, "relay_queue_item_rate"=>249.46666666666667, "relay_queue_items"=>0.0, "sync_queue_item_rate"=>0.0, "sync_queue_items"=>0.0, "work_queue_item_rate"=>187.1}
not_conn_endpoints []
num_conn_endpoints 3.0
num_endpoints 3.0
num_not_conn_endpoints 0.0
zones {"IZ-A"=>{"client_log_lag"=>0.0, "connected"=>true, "endpoints"=>["IcingaMaster", "IcingaMaster"], "parent_zone"=>"IZ-MASTER"}, "IZ-B"=>{"client_log_lag"=>0.0, "connected"=>true, "endpoints"=>["IcingaMaster"], "parent_zone"=>"IZ-MASTER"}, "IZ-MASTER"=>{"client_log_lag"=>0.0, "connected"=>true, "endpoints"=>["IcingaMaster"], "parent_zone"=>""}}
checker {"idle"=>32.0, "pending"=>0.0}
command 1.0
main-log 1.0
app {"enable_event_handlers"=>true, "enable_flapping"=>true, "enable_host_checks"=>true, "enable_notifications"=>true, "enable_perfdata"=>true, "enable_service_checks"=>true, "environment"=>"", "node_name"=>"IcingaMaster", "pid"=>7415.0, "program_start"=>1585774143.078506, "version"=>"2.11.2-1"}
ido-mysql {"connected"=>true, "instance_name"=>"default", "query_queue_item_rate"=>127.5, "query_queue_items"=>0.0, "version"=>"1.14.3"}
influxdb {"data_buffer_items"=>522.0, "work_queue_item_rate"=>62.483333333333334, "work_queue_items"=>0.0}
notification 1.0
Stats: [{"label"=>"Host checks/min", "value"=>295.0}, {"label"=>"Service checks/min", "value"=>3398.0}, {"label"=>"json_rpc queue rate", "value"=>"187.10"}, {"label"=>"ido-mysql queue rate", "value"=>"127.50"}, {"label"=>"influxdb queue rate", "value"=>"62.48"}]
Severity: [{"label"=>"360CON-3FE-R3-1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"JCLDCSRX - hostalive", "color"=>"red", "state"=>2}, {"label"=>"170CAL-PCI-PA500HA-1F1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"BELL529-PA7050HA-5F2 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"360CON-3FW-R3-1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"ARENA-2FW-R1-1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"24GEO-3FN-R2-2 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"ARENA-2FE-R0-2 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"360CON-1FW1-R1-2 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"133CAL-2F1--RE-N0 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"80BSTP-2FE-R1-1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"360CON-3FW-R2-2 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"BELL-3F-R1-1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"360CON-3FE-R1-1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"BELL418-PA850-VPN-4F1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"74GEO-1F-R1-1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"PPMAIN-3F-R1-1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"RHSC-3FW-R2-1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"RSS-1F-R2-1 - hostalive", "color"=>"red", "state"=>2}, {"label"=>"85PAT-BAT-1F1-R1-1 - hostalive", "color"=>"red", "state"=>2}]
App Info: {"icingaapplication"=>{"app"=>{"enable_event_handlers"=>true, "enable_flapping"=>true, "enable_host_checks"=>true, "enable_notifications"=>true, "enable_perfdata"=>true, "enable_service_checks"=>true, "environment"=>"", "node_name"=>"IcingaMaster", "pid"=>7415.0, "program_start"=>1585774143.078506, "version"=>"2.11.2-1"}}} Version: 2.11.2
Meter widget: Hosts 81.0/1131 Services 121.0/2196

Expected Behavior

Expecting dashing icinga2 to show data on front end immediately upon page refresh/load

Current Behavior

Dashing front end does not load data until 1m 55s after page refresh (so, it's succeeding, just after a long wait)
Significant to note here that the 10s scheduler interval was not changed

Possible Solution

Based on the log suggesting that the backend querying works, I imagine there's something on the front end referencing a request URL that's causing the data not to load in properly, but I could be completely wrong. Alternatively, there's a moving part linking the backend icinga2 job to the front end widgets that is malfunctioning due to the use of a non-local URL/address (proxy instead of localhost)

Steps to Reproduce (for bugs)

This is a link to our example of a malfunctioning dashing-icinga2 dashboard living behind a Microsoft App proxy: https://icinga-cofc.msappproxy.net/dashingicinga

Context

As the primary administrator of this application for the College of Charleston's IT department, my goal is to establish a reliable, nice dashboard to keep up on a shared office monitor that reveals monitoring data and alerts about the > 1000 hosts and > 3000 services that we are monitoring, from web server VMs to physical networking gear. It's really about closing the gap between problems and solutions by improving awareness.

Your Environment

• Dashing version (gem list --local dashing):

1.3.7

• Ruby version (ruby -V):

ruby 2.0.0p648 (2015-12-16) [x86_64-linux]

• Version of this project (tarball name, tag name or git show -1):

[root@RP-Icinga-M01 dashing-icinga2]# git show -1
commit 68cf8b254e0fefc3d5df547452a8b49477e54b9c
Merge: 5ce63b1 3a1d20c
Author: Michael Friedrich <michael.friedrich@gmail.com>
Date:   Wed Nov 27 11:00:48 2019 +0100

    Merge pull request #99 from mocdaniel/master

    Scrolling of crammed widgets

• Modifications to this project, if any (git diff):

[root@RP-Icinga-M01 dashing-icinga2]# git diff
diff --git a/assets/javascripts/application.coffee b/assets/javascripts/application.coffee
index 70fbd31..9f662e4 100644
--- a/assets/javascripts/application.coffee
+++ b/assets/javascripts/application.coffee
@@ -12,8 +12,8 @@ Dashing.on 'ready', ->
   Dashing.widget_base_dimensions ||= [300, 300]
   Dashing.numColumns ||= 5

-  contentWidth = (Dashing.widget_base_dimensions[0] + Dashing.widget_margins[0] * 2) * Dashing.n
-
+  contentWidth = "100%"
+  console.log("Dashing ready, setting batman immediate")
   Batman.setImmediate ->
     $('.gridster').width(contentWidth)
     $('.gridster ul:first').gridster
diff --git a/assets/stylesheets/application.scss b/assets/stylesheets/application.scss
index af74ec4..e1257f3 100644
--- a/assets/stylesheets/application.scss
+++ b/assets/stylesheets/application.scss
@@ -35,8 +35,8 @@ $icinga-blue: rgba(0, 149, 191, 1);
 // Base styles
 // ----------------------------------------------------------------------------
 html {
-  background-image: url("icinga-hero-background.jpg");
-  background-color: $background-color;
+  //background-image: url("icinga-hero-background.jpg");
+  background-color: black;
   /* Stretch the background image, don't duplicate it */
   -webkit-background-size: cover; /* For WebKit */
   -moz-background-size: cover;    /* Mozilla */
@@ -120,7 +120,7 @@ h3 {
 }

 #banner {
-  text-align: left;
+ // text-align: left;
   padding: 10px;
 }

@@ -172,12 +172,12 @@ h3 {
   display: flex;
   flex-direction: column;
   /* Pimp mode */
-  border-top: 1px solid rgba(255, 255, 255, 0.4);
-  border-bottom: 1px solid rgba(0, 0, 0, 0.4);
-  border-left: 1px solid rgba(255, 255, 255, 0.4);
-  border-right: 1px solid rgba(0, 0, 0, 0.4);
+  border-top: 2px solid #DDDDDD;
+  border-bottom: none;
+  border-left: 2px solid #DDDDDD;
+  border-right:none;
   box-shadow: 5px 5px 10px rgba(0, 0, 0, 0.4);
-  border-radius: 5px;
+  border-radius: 8px;
 }

 .header {
@@ -258,6 +258,7 @@ h3 {

 #container {
   display: inline-block;
+  width:100%;
 }


diff --git a/dashboards/icinga2.erb b/dashboards/icinga2.erb
index aa400af..0c02b45 100644
--- a/dashboards/icinga2.erb
+++ b/dashboards/icinga2.erb
@@ -25,7 +25,7 @@ $(function() {
         data-title="Hosts"
         data-labels="Up,Down"
         data-colornames="green,red"
-        data-datasets="20,13"
+        data-datasets="0,0"
         data-height="300"
         data-width="300"
       ></div>
@@ -38,7 +38,7 @@ $(function() {
         data-title="Services"
         data-labels="OK,Warning,Critical,Unknown"
         data-colornames="green,yellow,red,purple"
-        data-datasets="20,13,12,0"
+        data-datasets="0,0,0,0"
         data-height="300"
         data-width="300"
       ></div>
@@ -52,7 +52,7 @@ $(function() {
         data-title="Endpoints"
         data-labels="Connected,Not Connected"
         data-colornames="green,red"
-        data-datasets="42,404"
+        data-datasets="0,0"
         data-height="300"
         data-width="300"
       ></div>
@@ -71,7 +71,7 @@ $(function() {
         data-title="Active Checks"
         data-labels="Hosts/min,Services/min"
         data-colornames="aqua,lime"
-        data-datasets="42,404"
+        data-datasets="0,0"
         data-height="300"
         data-width="300"
       ></div>
@@ -86,7 +86,7 @@ $(function() {
         data-title="Downtimes"
         data-labels="Hosts,Services"
         data-colornames="blue,green"
-        data-datasets="42,404"
+        data-datasets="0,0"
         data-height="300"
         data-width="300"
       ></div>
@@ -101,7 +101,7 @@ $(function() {
         data-title="Acknowledgements"
         data-labels="Hosts,Services"
         data-colornames="blue,green"
-        data-datasets="42,404"
+        data-datasets="0,0"
         data-height="300"
         data-width="300"
       ></div>
@@ -140,10 +140,10 @@ $(function() {

     <!-- Icinga Web 2 iFrame. getIcingaWeb2Url() is defined in config.ru and reads from config/i
     <li data-row="3" data-col="1" data-sizex="2" data-sizey="2">
-      <div data-id="iframe" data-view="Iframe" data-title="Icinga Web Host Problems" data-url="<
+      <div data-id="iframe" data-view="Iframe" data-title="Icinga Web Host Problems" data-url="/
     </li>
     <li data-row="3" data-col="3" data-sizex="2" data-sizey="2">
-      <div data-id="iframe" data-view="Iframe" data-title="Icinga Web Service Problems" data-url
+      <div data-id="iframe" data-view="Iframe" data-title="Icinga Web Service Problems" data-url
     </li>
   </ul>

diff --git a/dashboards/layout.erb b/dashboards/layout.erb
index 2d4acb6..9e5873d 100644
--- a/dashboards/layout.erb
+++ b/dashboards/layout.erb
@@ -18,10 +18,16 @@
   <link rel="icon" href="/assets/favicon.png">

 </head>
-  <body>
+  <body style="width:100%; margin:10px auto; padding:5px">
     <div id="container">
       <div id="banner">
-        <img src="/assets/logo_icinga-inv.png">
+       <div style="width:48%; display:inline-block; text-align:left">
+               <img src="/assets/logo_icinga-inv.png">
+       </div>
+       <div style="width:48%; display:inline-block; text-align:right">
+               <img height="150" style="border-radius:15px" src="/assets/CofClogo.png">
+       </div>
+
       </div>
       <%= yield %>
     </div>
diff --git a/jobs/icinga2.rb b/jobs/icinga2.rb
index 3035823..317a3f1 100644
--- a/jobs/icinga2.rb
+++ b/jobs/icinga2.rb
@@ -20,7 +20,7 @@
 require './lib/icinga2'

 # initialize data provider
-icinga = Icinga2.new('config/icinga2.json') # fixed path
+icinga = Icinga2.new('config/icinga2.local.json') # fixed path

 SCHEDULER.every '10s', :first_in => 0 do |job|
   # run data provider
diff --git a/widgets/chartjs/chartjs.coffee b/widgets/chartjs/chartjs.coffee
index d37278b..a4b096c 100644
--- a/widgets/chartjs/chartjs.coffee
+++ b/widgets/chartjs/chartjs.coffee
@@ -43,6 +43,7 @@ class Dashing.Chartjs extends Dashing.Widget
     @colorNames = @get("colornames") && @get("colornames").split(",")

   ready: ->
+    console.log("Chart ready")
     Chart.defaults.global.defaultColor = 'rgb(255, 255, 255)'
     Chart.defaults.global.defaultFontColor = 'rgb(255, 255, 255)'
     Chart.defaults.global.legend.labels.fontColor = 'rgb(255, 255, 255)'
diff --git a/widgets/list/list.coffee b/widgets/list/list.coffee
index 0028073..2b8de12 100644
--- a/widgets/list/list.coffee
+++ b/widgets/list/list.coffee
@@ -1,5 +1,6 @@
 class Dashing.List extends Dashing.Widget
   ready: ->
+    console.log("List ready")
     if @get('unordered')
       $(@node).find('ol').remove()
     else

• Operating System and version:

[root@RP-Icinga-M01 dashing-icinga2]# cat /etc/os-release
NAME="Red Hat Enterprise Linux Server"
VERSION="7.7 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.7"
PRETTY_NAME="Red Hat Enterprise Linux"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.7:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.7
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.7"

• Client browser and version:

Chrome Version 80.0.3987.149 (Official Build) (64-bit)

Was also tested in Firefox but I don't have the version number as it was tested by a team member over a Zoom meeting

Thank you!

[mocdaniel]

Good morning,

to answer your first question, we use a working production version of dashing icinga behind a proxy with the master being located at let's say https://icinga2.subnet.localdomain and dashing being run in a docker container at https://monitoring.subnet.localdomain.

The end user in production accesses these webpages via a proxy and dashing delivers the SSE's via the proxy to the end user as well.

Apart from that, the server setup looks similar (we are using CentOS 7).

This being said, the end user is still within corporate intranet, so naturally there should be no high latencies.

From the logs I can't point straight at a possible error source, however I will try and look over your code changes. For now, my only guess would be that the server is somehow causing/getting too many requests from users, your icinga master etc, since this can sometimes lead to problems with ruby based evented applications.
Can you double check the time it takes dashing to send and process a request to your icinga master?

[dnsmichi]

I have never heard about msappproxy, Google explained that this is sort of a full blown proxy in Azure cloud - https://docs.microsoft.com/en-us/azure/active-directory/manage-apps/application-proxy-add-on-premises-application

I could imagine that the proxy intercepts network streams, and one of them is the event stream with Ruby and Server Side Events. Which are known to be buggy on Windows clients, and highly likely their support in their proxy is not intact either.

I would suggest asking the Azure support on whether they know if their proxy supports SSE.

Cannot access the URL with dashing atm, it says 503.

Keep in mind though that Dashing and the underlaying Ruby framework using old web technologies, and not something like web sockets. At some point in time, you'll need to find a modern replacement for Dashing.

In terms of the diffs:

-icinga = Icinga2.new('config/icinga2.json') # fixed path
+icinga = Icinga2.new('config/icinga2.local.json') # fixed path

is not necessary, the library code ensures to load the correct .local file when it is there.

[austinjhunt]

@mocdaniel @dnsmichi Thank you both so much for the timely responses. I wasn't aware of a potential issue with SSE but I'll do some research to see if that's supported.

As far as a high request load, I'm actually currently the only person consistently accessing the web server, I had one other user testing it over a Zoom meeting with Firefox but aside from that its visibility is pretty hidden right now.

Also, interestingly, @mocdaniel I did just run the stopwatch again (specifically, started the foreground service, waited about 5 seconds for backend service to initialize, entered the proxy URL in the browser and immediately started the stopwatch), and the time was essentially exactly the same, 1m56s instead of 1m55s.

And my apologies, I forgot that I had terminated the service running in the foreground without starting it back up in the background before providing the URL, here it is again, it's now running as a background service: https://icinga-cofc.msappproxy.net/dashingicinga

Again, thank you both!

[dnsmichi]

Just a guess, what happens if you disable the Icinga Web 2 Iframe loading at the bottom (edit the dashboard and comment it out). It seems this takes quite a while.

[austinjhunt]

Just commented out both of the bottom iFrames, re-ran the timer, got the same 1m55s rendering time for the data. Thank you for pointing that out, though!

[dnsmichi]

Can you analyse in your browser's dev console which portions take long to load and then render? This trace would truly be interesting to learn more about the problem.

[mocdaniel]

@austinjhunt I didn't refer to the request load from the user's side, but the amount of requests dashing-icinga is issuing towards your master each refresh interval. If you have too many separate requests going off this might delay the response a user of the web-app might get. Still, 1.9 minutes (this is what I get when requesting https://icinga-cofc.msappproxy.net/events) is oddly high.

Nonetheless, please check your icinga2 logs (should be on /var/log/icinga2/icinga2.log on your RHEL server) explicitly for all requests arriving during 1 refresh interval of your icingaweb instance.
For easier identifying, you could run dashing icinga in the context of an easily identifiable api user, since his credentials are sent and logged with the requests.

In addition, maybe measure the time one single request from dashing to the master takes to return data, you could use the api methods shipped with dashing for that purpose.

[austinjhunt]

Can you analyse in your browser's dev console which portions take long to load and then render? This trace would truly be interesting to learn more about the problem.

@dnsmichi I exported an HAR file for the request to the proxy, which again took an exact 1m55s to load initial data into front end, super interesting. Can't attach the HAR file here, unfortunately, as that file type isn't supported for attachment.
Here's a Google Drive link to the file: https://drive.google.com/drive/folders/1hv_CMOnyvzS0wnjcTgCw5HCqxTHNLRsd?usp=sharing
Here is a screenshot of the console with most of the field visibility enabled:

/events is definitely the bottleneck.

Nonetheless, please check your icinga2 logs (should be on /var/log/icinga2/icinga2.log on your RHEL server) explicitly for all requests arriving during 1 refresh interval of your icingaweb instance.
@mocdaniel

One refresh interval in icinga2.log filtering by the dashing API user:

[root@RP-Icinga-M01 ~]# tail -f /var/log/icinga2/icinga2.log | grep dashing
[2020-04-03 12:44:42 -0400] information/HttpServerConnection: Request: GET /v1/status (from [127.0.0.1]:52792), user: dashing, agent: rest-client/2.1.0 (linux-gnu x86_64) ruby/2.0.0p648).
[2020-04-03 12:44:52 -0400] information/HttpServerConnection: Request: GET /v1/status/IcingaApplication (from [127.0.0.1]:52846), user: dashing, agent: rest-client/2.1.0 (linux-gnu x86_64) ruby/2.0.0p648).
[2020-04-03 12:44:52 -0400] information/HttpServerConnection: Request: GET /v1/status/CIB (from [127.0.0.1]:52850), user: dashing, agent: rest-client/2.1.0 (linux-gnu x86_64) ruby/2.0.0p648).
[2020-04-03 12:44:52 -0400] information/HttpServerConnection: Request: GET /v1/objects/hosts (from [127.0.0.1]:52854), user: dashing, agent: rest-client/2.1.0 (linux-gnu x86_64) ruby/2.0.0p648).
[2020-04-03 12:44:52 -0400] information/HttpServerConnection: Request: GET /v1/objects/services (from [127.0.0.1]:52858), user: dashing, agent: rest-client/2.1.0 (linux-gnu x86_64) ruby/2.0.0p648).
[2020-04-03 12:44:52 -0400] information/HttpServerConnection: Request: GET /v1/status (from [127.0.0.1]:52862), user: dashing, agent: rest-client/2.1.0 (linux-gnu x86_64) ruby/2.0.0p648).

As for the amount of time it takes for a request from dashing to the master, (not sure if this was the proper way of identifying that, but it seemed logical - not too familiar with Ruby) I calculated the elapsed time from the moment right before to right after the icinga.run line executes in the scheduled icinga2.rb job. Averaging less than a second, definitely not the 1.9m. Which leads me to think it is indeed the send_event (one or all) methods that are causing the delay. The only question is, why is it a constant delay and not variable?

I really appreciate both of your feedback (feedbacks?). Let me know if you have any additional thoughts. It does look like Azure is claiming their proxy handles SSE but there's a chance it's messing up on their end.

[dnsmichi]

I did not do much with thin or Ruby as server in general, the only thing I do know is that it is implemented as single thread pattern, and whenever send_event is called it needs to wait up until the queue is empty to continue. There might be something which blocks the server for 2 minutes (or 1.9m).

To nail it down further, try to comment out all the object getters, and only use one send_event function (a simple one without API interaction towards icinga2). Does this change anything?

[austinjhunt]

Wow! Okay. Very interesting results. I commented out everything except the very first send_event function (for "icinga-stats"), and now in Chrome's network tab, the /events resource is taking anywhere from 2.8 minutes to 3.1 minutes to load. No longer a constant 1.9 minutes. Adding some screenshots. I'm a bit confused about why ALL the event data is being pulled if the only one not commented out is icinga-stats, though. I would expect that cell to be the only one with content since nothing else is getting sent from the server. I guess that means I didn't comment out the send_event functions from the right file?

2.8 minutes here:

3.0 minutes here:

3.1 minutes here:

[mocdaniel]

Hi @austinjhunt and sorry for the long time without reply, we were occupied by getting the repository transfer done as smoothly as possible.

Commenting out the send_event block should keep the server from pushing the data to your webclient, anyways, the information gets pulled by the server from your Icinga2 instance before that already, because all the data gets queried when icinga.run gets executed in line 27 of icinga2.rb.

To me, this is a strong indicator that the proxy is the problem, given that you already measured the time needed for dashing and Icinga2 to communicate.

Also, I just reread your initial post and the fact that there is no delay when you access the dashboard via the server's FQDN directly sounds like proxy problems as well.

I will look around Microsoft's forums for a bit longer but I am afraid there's nothing on our end we can do to resolve your issues.