Add concept of Sensor Centre

cookbook/sections/20250327_Sensor_Centre.adoc

@@ -0,0 +1,159 @@
+= Sensor Centres in WIS 2.0
+:toc: macro
+:sectnums: all
+:version: 0.1
+:author: Rémy Giraud
+:email: remy@giraud.me
+:revnumber: 0.1
+:revdate: 16.02.2025
+
+<<<
+
+Introduction::
+
+
+The documentation _Sensor Centres in WIS 2.0_ explains the *concept* of Sensor Centres in WIS2.0 and gives some examples of such tools, that are currently available 
+or could be added, if volunteers WIS Members are willing to work on this.
+
+As describe in the Manual of WIS, Volume II, WIS 2.0 as a collective IT system needs monitoring. Unlike typical integrated IT systems under 
+a single management authority, it was not possible to us a tool like Zabbix, which is by nature fairly intrusive toward the system it monitors. 
+Therefore, a solution where each component of the system provides information on its behaviour was preferred.
+
+It has been decided to use openmetrics, and primarily two additional open source tools that can collect (prometheus) and visualize (grafana) the metrics.
+
+At the moment, and for three Global Services (Broker, Cache and Discovery Catalogue), a set of metrics have been defined.
+Each instance of those Global Services must provide the agreed metrics.
+
+Then, the Global Monitor collect the metrics and present a set of Grafana Dashboard.
+
+This monitoring architecture is by design, extendable.
+
+When new metrics are defined, it is possible to collect them and to visualize them.
+
+To go beyond the metrics defined for the Global Services, the concept of Sensor Centre has been introduced.
+
+A Sensor Centre relies on WIS 2.0 Global Services and potentially WIS2 Nodes. It can implement further processing on any part of the WIS 2.0 architecture
+
+. receiving Notification Messages
+. downloading files
+. processing the content of the files
+. ...
+
+the result of this processing is a specific set of _new_ metrics, that will be in turn collected either by Global Monitor 
+or by additional monitoring systems for statistical analysis, visualization,...
+
+It must be noted though that WIS 2.0 ecosystem *does not* provide any Sensor Centre. 
+
+It provides the tooling to implemented Sensor Centre by taking advantages of the monitoring solution of WIS 2.0. It is the responsibility of 
+a communiy relying on WIS 2.0 for its data exchange to implement and to eventually deploy Sensor Centre(s) to monitor whatever seems appropriate to monitor its operations.
+
+image:2025-02-22T10-53-08-434Z.png[] 
+
+The rest of this document provide example of Sensor Centre
+
+Comparing the behaviour of Global Cache(s)::
+
+In WIS 2.0 each Global Cache is independent of the other Global Caches. According to the specification of WIS 2.0, Global Cache (see https://wmo-im.github.io/wis2-guide/guide/wis2-guide-APPROVED.html#_2_7_4_1_technical_considerations ) :
+
+`Global Caches will operate independently of one another. Each Global Cache will hold a full copy of the cache – although there may be small differences between the various Global Caches as data availability notification messages propagate through WIS to each one. There is no formal synchronization between Global Caches.`
+
+it is therefore interesting to verify, for example:
+
+. what is the average delay to cache the data made available by WIS2 Node
+. are all files published as _core data_ (and with `cache: true` in the Notification Message) by WIS2 Nodes are effectively available in all Global Caches
+. are files missed by some Global Cache or more critically by all Global Caches
+
+Such metrics would provide useful information to identify problems, to help Global Caches to fix them, to define KPI that could be used to objectively measure the effective performance of the Cache.
+
+It could also be used to detect anomalies from the WIS2 node, such as the reuse too frequently of the same `data_id`.
+
+It is agreed to call this type of Sensor Centre: sensor-global-cache.
+The full centre-id will therefore be 2 letter country code - name of the centre - sensor-global-cache
+
+For such a centre operated by Météo-France the centre-id would be fr-meteofrance-sensor-global-cache
+
+A list of metrics has been defined for this Sensor Centre (see https://github.com/wmo-im/wis2-metric-hierarchy/blob/main/metrics/sgc.csv):
+
+[cols="3*", options="header"]
+|=============================================================================================================================================================
+| Name                                     | Labels                          | Description                                                                    
+| wmo_wis2_sgc_cache_delay_seconds         | globalcache,centre_id,report_by | Delay between origin and cache message                                         
+| wmo_wis2_sgc_messages_cached_total       | globalcache,centre_id,report_by | Number of data files cached for centre_id                                      
+| wmo_wis2_sgc_messages_cached_delay_total | globalcache,centre_id,report_by | Number of data files cached for centre_id within defined delay (120s 300s 600s)
+| wmo_wis2_sgc_messages_published_total    | centre_id,report_by             | Number of cacheable data files published                                       
+| wmo_wis2_sgc_messages_missed_total       | globalcache,centre_id,report_by | Number of cacheable data not in global cache                                   
+| wmo_wis2_sgc_messages_missed_all_total   | centre_id,report_by             | Number of cacheable data not in any cache                                      
+|=============================================================================================================================================================
+
+The processing of this Sensor Centre is as follow:
+
+- Subscribe to a given `origin/a/wis2/...`, it could be `#` or a particular centre-id on at least one Global Broker
+- Subscribe to a given `cache/a/wis2/...`, it could be `#` or a particular centre-id on at least one Global Cache - The subscription must be done on the broker of the Global Cache (unlike normal subscription to be made only on the Global Broker)
+
+Only on the subscription to the Global Broker:
+
+- Discard the Notification Message if is it `recommended` data or if `cache: false` as the data will not be cached
+- Detect any duplicates `data_id` not including a `rel: update` within a period of at least X hours
+- Store the time where the message as been received
+- (optional) Store the full Notification Message - this can be useful to analyse systematic issues
+
+For each of the subscription to the various Global Caches:
+
+- For each Notification Message and using `data_id`, make the difference between the time was received on `origin` and the same `data_id` on `cache`: Time~Cache~ - Time~Origin~
+- Update the `wmo_wis2_sgc_cache_delay_seconds` metric with this value
+- Compare this value with the three threshold defined in the matric table above. Increase by 1 `wmo_wis2_sgc_messages_cached_delay_total` using the threshold as a label (so less than 120s, less that 300s, less than 600s)
+- If no Notification Message for the `data_id` is received after the highest threshould (here 600s), increase by 1 `wmo_wis2_sgc_messages_missed_total`
+
+
+If no Global Cache has cached the data, increase by 1 `wmo_wis2_sgc_messages_missed_all_total`
+
+All the metrics must be exposed for scraping by the Global Monitor.
+
+If desirable and in order to further analyse the situation, the origin Notification Message can be published on monitor/a/wis2/centre-id sensor centre/centre-id of the originator of the message.
+
+Comparing the behaviour of Global Brokers::
+
+By design, all Notification Messages must be avaimable on all Global Brokers. Either after being received directly from the source centre-id or indirectly from another Global Broker.
+
+During the validation tests ran in autumn 2024, it was check that for a (small) giver number of Notification Messages all Global Brokers were behaving as expected.
+
+However, as a complement or as a way to detect anomalies, it could be useful to effectively compare, using operational Notification Messages that all Notification Messages are available on all Global Broker.
+
+It is expected that the Global Brokers will be _almost_ in sync, and the delay between having the same `ìd` on all Global Broker will be less than 15 secondes.
+
+This type of Sensor Centre can be called: sensor-global-broker.
+The full centre-id will therefore be 2 letter country code - name of the centre - sensor-global-broker.
+
+
+[cols="3*", options="header"]
+|=============================================================================================================================================================
+| Name                                     | Labels                          | Description                                                                    
+| wmo_wis2_sgb_missed_total        | globalbroker,centre_id,report_by | Number of Notification Messages missed by the Global Broker                                                                                
+|=============================================================================================================================================================
+_to be further expanded_
+
+The processing of this Sensor Centre is as follow:
+
+- Subsbribe to `origin/a/wis2/...` and `cache/a/wis2/...`, it could be `#` or a particular centre-id on at all Global Brokers
+- For each `id` received, check if the `id` is received by all Global Brokers within the 15s time window
+
+Conclusion::
+
+This document presents the concept of Sensor Centre and provide two examples of such tools.
+
+Obviously, many more types of Sensor Centre can be designed.
+
+Each community within WIS2.0 can design Sensor Centre tailored to its needs.
+
+The approach will always be similar:
+
+. Discuss the opportunity of developping a Sensor Centre to assess how the centre-id providing the data, or how the Global Services are performing, or anything relying on WIS 2.0 for addressing the needs of the community
+. Agree on a list of metrics than can be implemented to perform the assessment
+. Register the list of metrics in the WMO metrics repository  https://github.com/wmo-im/wis2-metric-hierarchy/
+. Develop the Sensor Centre 
+. Operate one or more instance of the Sensor Centre
+. Register the Sensor(s) Centre centre-id in the WMO Register
+. Ensure that the metrics are correctly scraped by the Global Monitor
+. Provide the Grafana dashboard that the Global Monitor will host
+
+It is also possible for item 7. and 8. above to use another Monitor Centre if preferred by the community.

cookbook/sections/WIS 2.0 node - User requirements specifications.adoc

@@ -0,0 +1,182 @@
+= WIS 2.0 node - User requirements specifications
+:toc:
+:toclevels: 2
+:toc-title: Table of Contents
+
+
+== Introduction
+
+=== Applicable Documents
+
+[width="100%",cols="20%,20%,60%",options="header",]
+|===
+| |*Document Title* |*Reference*
+|AD-0 a|
+Manual on WMO Information System Volume II. WMO Information System 2.0
+| https://library.wmo.int/records/item/68731-manual-on-the-wmo-information-system-volume-ii-wmo-information-system-2-0
+|===
+
+=== Reference Documents
+
+[width="100%",cols="20%,20%,60%",options="header",]
+|===
+| |*Document Title* |*Reference*
+|RD-0 |WMO Information System 2.0 Guide |https://library.wmo.int/records/item/69130-guide-to-the-wmo-information-system-volume-ii
+|RD-1 |WMO Information System 2.0 Strategy |https://library.wmo.int/doc_num.php?explnum_id=4620
+|RD-2 |WIS 2.0: How to define successful transition? |https://wmo-teams.atlassian.net/wiki/spaces/WIS2/pages/301957121/WIS2.0+how+to+define+successful+Transition
+|===
+
+== System overview
+
+=== System Context
+
+The WIS 2.0 node will be available to receive data from the upstream data production system, and to serve these data via the internet to the WIS 2.0 environment.
+
+The WIS 2.0 node will be available to receive metadata and to serve these metadata via the internet to the WIS 2.0 environment.
+
+WIS 2.0 Global Brokers will subscribe to the WIS 2.0 node, and will receive publication messages when data and metadata become available.
+
+*WIS 2.0 node*
+
+. Selected core and recommended data from upstream systems
+
+. Notification messages to WIS 2.0 Global Brokers
+
+. Selected core data to WIS 2.0 Global Caches
+
+. Subscription messages from WIS 2.0 Global Brokers
+
+. Remaining core data and recommended data to WIS 2.0 end users
+
+[[IDEF0_Context_Diagram.jpg]]
+.IDEF0 Context Diagram
+image::IDEF0_Context_Diagram.jpg[Context Diagram, width=400, height=300]
+
+=== External Interfaces
+
+As identified in the context diagram <<IDEF0 Context Diagram>>, the WIS 2.0 node will have external interfaces with WIS 2.0 Global Brokers, and WIS 2.0 Global Caches. In each case, the communications will be via the internet.
+
+The Global Brokers will subscribe to the MQTT broker on the node. This will be done using the standard MQTT secure port, 8883.
+
+The Global Caches will retrieve core data from the node by accessing the HTTP server on the node.
+
+End users will retrieve core data not retained by the Global Caches from the node by accessing the HTTP server on the node. End users will also retrieve recommended data from the node by accessing the HTTP server on the node, subject to appropriate access control.
+
+=== Concepts and Constraints
+
+The WIS 2.0 node will be compliant with the Manual on WMO Information System Volume II. WMO Information System 2.0 ++[++AD-0++]++.
+
+== User requirements
+
+The purpose of the WIS 2.0 node (hereafter, referred to as the Node) is to be available to receive data from the production system, and to serve these data via the internet to the WIS 2.0 environment.
+
+In order to reach this target, the following high-level user requirements need to be fulfilled:
+
+*_USR-0001_*
+
+The Node shall comply with the specification given in the Manual on WMO Information System Volume II. WMO Information System 2.0, ++[++AD-0++]++.
+
+*_USR-0101_*
+
+The WIS 2.0 node shall include an MQTT broker, using MQTT 5.0 (hereafter, referred to as the Broker).
+
+*_USR-0150_*
+
+The Broker included in the Node shall be accessible via MQTT protocol over the internet.
+
+*_USR-0201_*
+
+The Node will allow subscriptions from the WIS 2.0 Global Brokers.
+
+*_USR-0210_*
+
+The Node will restrict subscriptions to the Broker by only WIS 2.0 Global Brokers by filtering their incoming IP addresses as made available by WMO Secretariat.
+
+*_USR-0220_*
+
+Access to the Broker shall be password controlled.
+
+*_USR-230_*
+
+The secure version of MQTT (MQTTS) shall be used. The use of SSL certificates to support this shall be maintained over the lifetime of the system.
+
+*_USR-240_*
+
+The Broker shall publish messages using MQTT’s Quality of Service (QoS) level 1. This is defined as follows: “_The broker/client will deliver the message at least once, with confirmation required._”.
+
+*_USR-0301_*
+
+Upon the arrival of data for distribution via WIS 2.0, the Broker shall publish an MQTT message announcing the availability of the data.
+
+*_USR-0320_*
+
+Notification messages published by the Node shall be formatted in geoJSON, in accordance with the Manual on WMO Information System Volume II. WMO Information System 2.0, ++[++AD-0++]++.
+
+*_USR-0340_*
+
+Notification messages published by the Node shall be published using an MQTT topic defined in accordance with the WIS 2.0 topic hierarchy.
+
+*_USR-0360_*
+
+The MQTT topic used in notification messages shall not be configured to retain messages.
+
+*_USR-0401_*
+
+The Node shall provide access to core data via HTTP over the internet.
+
+*_USR-0450_*
+
+The Node shall provide access to recommended data via HTTP over the internet, subject to appropriate access control.
+
+*_USR-0500_*
+
+The Node shall provide access to core data via HTTPS over the internet.
+
+*_USR-0550_*
+
+The Node shall provide access to recommended data via HTTPS over the internet, subject to appropriate access control.
+
+*_USR-0601_*
+
+The Node shall support the retrieval core data by the WIS 2.0 Global Caches.
+
+
+=== Extract from the Manual on WMO Information System Volume II. WMO Information System 2.0 ++[++AD-0++]++:
+
+FUNCTIONAL REQUIREMENTS OF A WIS NODE
+
+3.6.1 General
+
+3.6.1.1 A WIS node is the component that enables an NC or DCPC to publish their data and discovery metadata via WIS.
+
+3.6.1.2 See also 3.3 (Functional requirements of an NC) and 3.4 (Functional requirements of a DCPC).
+
+3.6.2 Provide access to data and discovery metadata
+
+3.6.2.1 A WIS node shall provide access to data in accordance with the WMO Unified Data Policy (Resolution 1 (Cg-Ext-2021)).
+
+3.6.2.2 A WIS node shall allow one or more Global Caches to access and download core data it publishes for real-time and near real-time exchange. Global Caches provide highly available access to copies of these resources.
+
+3.6.2.3 A WIS node may restrict access to its core data, relying on Global Caches providing access to data consumers.
+
+3.6.2.4 A WIS node may provide access to data using a Web-based Application Programming Interface (API).
+
+3.6.2.5 A WIS node shall provide access to discovery metadata describing the data it makes available and how that data can be accessed. Discovery metadata from a WIS node is added to the Global Discovery Catalogue to create a consolidated view of data available from all WIS nodes.
+
+3.6.2.6 A WIS node shall have the capability to publish notifications via a Message Broker.
+
+3.6.2.7 A WIS node shall publish notifications via its Message Broker about updates to the data and discovery metadata it provides – including the availability of new data, changes to discovery metadata, and removal of a data set from WIS.
+
+3.6.2.8 A WIS node shall use a standardized topic structure when publishing notifications. Note: More information on the standardized topic structure is provided in the Guidance on technical specifications of WIS 2.0.
+
+3.6.2.9 A WIS node shall allow one or more Global Brokers to subscribe to notifications published via its Message Broker. Global Brokers provide highly available distribution of notifications published by a WIS node.
+
+3.6.2.10 See also 4.3 (WIS-TechSpec-2: Publishing data and discovery metadata).
+
+Note: More information on the function and implementation of a WIS node is provided in the Guidance on technical specifications of WIS 2.0.
+
+3.6.3 Monitor performance of a WIS node
+
+3.6.3.1 Each WIS node shall contribute to monitoring the performance of WIS.
+
+3.6.3.2 See also 4.7 (WIS-TechSpec-6: Managing operations of the WIS).