Air quality agent update

Agents/AirQualityAgent/Dockerfile

@@ -2,7 +2,7 @@
 #==================================================================================================
 
 # Reference published Docker image for Stack-Client resources to use
-FROM docker.cmclinnovations.com/stack-client:1.6.2 as stackclients
+FROM ghcr.io/cambridge-cares/stack-client:1.40.1 as stackclients
 
 #------------------------------------------------------
 # Base image to be reused

Agents/AirQualityAgent/README.md

@@ -33,11 +33,7 @@ docker login ghcr.io -u <github_username>
 <github_personal_access_token>
 ```
 
-### **3) Accessing CMCL docker registry**
-
-The agent requires building the [Stack-Clients] resource from a Docker image published at the CMCL docker registry. In case you don't have credentials for that, please email `support<at>cmclinnovations.com` with the subject `Docker registry access`. Further information can be found at the [CMCL Docker Registry] wiki page.
-
-### **4) VS Code specifics**
+### **3) VS Code specifics**
 
 In order to avoid potential launching issues using the provided `tasks.json` shell commands, please ensure the `augustocdias.tasks-shell-input` plugin is installed.
 
@@ -59,10 +55,12 @@ bash ./stack.sh remove <STACK_NAME> -v
 
 After spinning up the stack, the GUI endpoints to the running containers can be accessed via Browser (i.e. adminer, blazegraph, ontop, geoserver). The endpoints and required log-in settings can be found in the [spin up the stack] readme.
 
+Alternatively, you may copy [airqualityagent.json](stack-manager-config/inputs/config/services/airqualityagent.json) in the [stack-manager config directory](https://github.com/cambridge-cares/TheWorldAvatar/tree/main/Deploy/stacks/dynamic/stack-manager/inputs/config/services). Note that the Air Quality Agent is now exposed on localhost port 3838 instead.
+
 &nbsp;
 ## 1.3 Deploying the agent to the stack
 
-This agent requires [JPS_BASE_LIB] and [Stack-Clients] to be wrapped by [py4jps]. Therefore, after installation of all required packages (incl. `py4jps >= 1.0.30`), the `StackClients` resource needs to be added to allow for access through `py4jps`. All required steps are detailed in the [py4jps] documentation. However, the following information should suffice in this context:
+This agent requires [JPS_BASE_LIB] and [Stack-Clients] to be wrapped by [py4jps]. Therefore, after installation of all required packages (incl. `py4jps >= 1.0.38`), the `StackClients` resource needs to be added to allow for access through `py4jps`. All required steps are detailed in the [py4jps] documentation. However, the following information should suffice in this context:
 * When building the Docker images, the `StackClients` resource is copied from the published Docker image (details in the [Dockerfile])
 * For testing purposes, the latest `StackClients` resource needs to be compiled and installed locally using the [py4jps] Resource Manager
 
@@ -77,7 +75,7 @@ bash ./stack.sh build
 bash ./stack.sh start <STACK_NAME>
 ```
 
-In case of time out issues in automatically building the StackClients resource, please try pulling the required stack-clients image first by `docker pull docker.cmclinnovations.com/stack-client:1.6.2`
+In case of time out issues in automatically building the StackClients resource, please try pulling the required stack-clients image first by `docker pull ghcr.io/cambridge-cares/stack-client:1.40.1`
 
 The *debug version* will run when built and launched through the provided VS Code `launch.json` configurations:
 > **Build and Debug**: Build Debug Docker image (incl. pushing to [Github container registry]) and deploy as new container (incl. creation of new `.vscode/port.txt` file)
@@ -135,10 +133,13 @@ Agent start-up will automatically register a recurring task to assimilate latest
 - GET request to update all UK-AIR stations and associated readings, and add latest data for all time series (i.e. instantiate missing stations and readings and append latest time series readings)
 > `/airqualityagent/update/all`
 - GET request to retrieve data about UK-AIR stations and create respective JSON output files (i.e. request expects all individual query parameter to be provided in a single nested JSON object with key 'query') - **please note** that this query was required to create DTVF input files previously and is now deprecated as DTVF retrieves visualisation input from PostGIS via Geoserver. This endpoint is mainly kept here for reference purposes.
-> `/api/metofficeagent/retrieve/all`
+> `/airqualityagent/retrieve/all`
 
 Example requests are provided in the [resources] folder, which also contain further information about allowed parameters.
 
+## Note
+
+It has been observed that the UK-AIR API could be unstable, in particular for time series updates. If you receive an error message after a call to the agent, please try again later.
 
 &nbsp;
 # 3. Agent Tests

Agents/AirQualityAgent/agent/datainstantiation/readings.py

@@ -515,6 +515,22 @@ def retrieve_timeseries_information_from_api(ts_ids=[], days_back=7) -> dict:
 
     return infos
 
+def repeat_http_post(url, body, headers, max_try = 3):
+    """
+    This will send a HTTP POST request and re-try if it fails.
+    Currently only used for time series retrieval as it is found to be unstable.
+    """
+    counter = 0
+    while counter < max_try:
+        try:
+            r = requests.post(url=url, data=json.dumps(body), headers=headers)
+            # Extract time series data
+            return r.json()
+        except Exception as ex:
+            logger.warning(f"Error while retrieving time series data from API: {ex}")
+            counter = counter + 1
+            time.sleep(5)
+    raise APIException("Error while retrieving time series data from API. Please try again later.")
 
 def retrieve_timeseries_data_from_api(crs: str = 'EPSG:4326', ts_ids=[], 
                                       period='P1D', chunksize=25) -> dict:
@@ -566,13 +582,7 @@ def retrieve_timeseries_data_from_api(crs: str = 'EPSG:4326', ts_ids=[],
                 "timeseries": chunk
         }
 
-        try:
-            r = requests.post(url=url, data=json.dumps(body), headers=headers)
-            # Extract time series data
-            ts_data = r.json()
-        except Exception as ex:
-            logger.error(f"Error while retrieving time series data from API: {ex}")
-            raise APIException("Error while retrieving time series data from API.") from ex
+        ts_data = repeat_http_post(url, body, headers)
 
         df = pd.DataFrame.from_dict(ts_data, orient='index')
         # Remove rows without entries

Agents/AirQualityAgent/agent/kgutils/stackclients.py

@@ -40,7 +40,6 @@ class StackClient:
 class OntopClient(StackClient):
 
     def __init__(self, query_endpoint=ONTOP_URL):
-        # Initialise OntopClient as RemoteStoreClient
         try:
             self.ontop_client = self.jpsBaseLib_view.RemoteStoreClient(query_endpoint)
         except Exception as ex:
@@ -70,7 +69,8 @@ def upload_ontop_mapping():
             f = OntopClient.stackClients_view.java.io.File(ONTOP_FILE)
             fp = f.toPath()
             # Update ONTOP mapping (requires JAVA path object)
-            OntopClient.stackClients_view.OntopClient().updateOBDA(fp)
+            ontop_client_instance = OntopClient.stackClients_view.OntopClient.getInstance()
+            ontop_client_instance.updateOBDA(fp)
         except Exception as ex:
             logger.error("Unable to update OBDA mapping.")
             raise StackException("Unable to update OBDA mapping.") from ex
@@ -185,7 +185,7 @@ class GdalClient(StackClient):
     def __init__(self):
         # Initialise GdalClient with default upload/conversion settings
         try:
-            self.client = self.stackClients_view.GDALClient()
+            self.client = self.stackClients_view.GDALClient.getInstance()
             self.orgoptions = self.stackClients_view.Ogr2OgrOptions()
         except Exception as ex:
             logger.error("Unable to initialise GdalClient.")
@@ -196,7 +196,7 @@ def uploadGeoJSON(self, geojson_string, database=DATABASE, table=LAYERNAME):
         """
             Calls StackClient function with default upload settings
         """
-        self.client.uploadVectorStringToPostGIS(database, table, geojson_string,
+        self.client.uploadVectorStringToPostGIS(database, "public", table, geojson_string,
                                                 self.orgoptions, True)
 
 
@@ -206,7 +206,7 @@ def __init__(self):
 
         # Initialise Geoserver with default settings
         try:
-            self.client = self.stackClients_view.GeoServerClient()
+            self.client = self.stackClients_view.GeoServerClient.getInstance()
             self.vectorsettings = self.stackClients_view.GeoServerVectorSettings()
         except Exception as ex:
             logger.error("Unable to initialise GeoServerClient.")
@@ -225,7 +225,7 @@ def create_postgis_layer(self, geoserver_workspace=GEOSERVER_WORKSPACE,
             Please note: Postgis database table assumed to have same name as
                          Geoserver layer
         """
-        self.client.createPostGISLayer(None, geoserver_workspace, postgis_database,
+        self.client.createPostGISLayer(geoserver_workspace, postgis_database, "public",
                                        geoserver_layer, self.vectorsettings)
 
 

Agents/AirQualityAgent/agent/utils/stack_configs.py

@@ -38,7 +38,7 @@ def retrieve_settings():
     pg = stackClientsView.PostGISEndpointConfig("","","","","")
     pg_conf = containerClient.readEndpointConfig("postgis", pg.getClass())
     # Ontop
-    ont = stackClientsView.OntopEndpointConfig("","","","","")
+    ont = stackClientsView.OntopEndpointConfig("","","","")
     ont_conf = containerClient.readEndpointConfig("ontop", ont.getClass())
 
     # Extract PostgreSQL/PostGIS database URL

Agents/AirQualityAgent/app_entry_point.sh

@@ -1,3 +1,3 @@
 #!/bin/bash
 # timeout set to 120min to avoid exceptions for long API/KG calls
-gunicorn --bind 0.0.0.0:5000 agent.flaskapp.wsgi:app --timeout 7200
+gunicorn --bind 0.0.0.0:5000 agent.flaskapp.wsgi:app --timeout 7200 --preload

Agents/AirQualityAgent/docker-compose.yml

@@ -2,7 +2,7 @@ version: "3.8"
 
 services:  
   airquality_agent:
-    image: ghcr.io/cambridge-cares/airquality_agent:1.0.0
+    image: ghcr.io/cambridge-cares/airquality_agent:1.1.0-SNAPSHOT
     environment:
       - STACK_NAME=${STACK_NAME}
       # Target Blazegraph namespace

Agents/AirQualityAgent/setup.py

@@ -16,9 +16,11 @@
     install_requires= [
         'APScheduler==3.9.1',
         'Flask==2.2.2',
+        'werkzeug==2.2.2',
         'JayDeBeApi==1.2.3',
         'pandas~=1.4',
-        'py4jps==1.0.34', 
+        'numpy~=1.24',
+        'py4jps==1.0.38', 
         'requests==2.28.1',
     ]
 )

Agents/AirQualityAgent/stack-manager-config/inputs/config/services/airqualityagent.json

@@ -0,0 +1,59 @@
+{
+    "ServiceSpec": {
+        "Name": "air-quality-agent",
+        "TaskTemplate": {
+            "ContainerSpec": {
+                "Image": "ghcr.io/cambridge-cares/airquality_agent:1.1.0-SNAPSHOT",
+                "Env": [
+                    "STACK_NAME=${STACK_NAME}",
+                    "NAMESPACE=airquality",
+                    "LAYERNAME=airquality",
+                    "DATABASE=postgres",
+                    "GEOSERVER_WORKSPACE=stations",
+					"ONTOP_FILE=/app/resources/ontop.obda"
+                ],
+                "Mounts": [
+                    {
+                        "Type": "bind",
+                        "Source": "../../../../../../Agents/AirQualityAgent/output",
+                        "Target": "/app/output"
+                    },
+                    {
+                        "Type": "volume",
+                        "Source": "logs",
+                        "Target": "/root/.jps"
+                    }
+                ],
+				"Configs": [
+                    {
+                        "ConfigName": "blazegraph"
+                    },
+					{
+                        "ConfigName": "geoserver"
+                    },
+					{
+                        "ConfigName": "ontop"
+                    },
+					{
+                        "ConfigName": "postgis"
+                    }
+                ],
+                "Secrets": [
+                    {
+                        "SecretName": "postgis_password"
+                    },
+					{
+                        "SecretName": "geoserver_password"
+                    }
+
+                ]
+            }
+        }
+    },
+    "endpoints": {
+        "ui": {
+            "url": "http://localhost:5000/airqualityagent/",
+            "externalPath": "/airqualityagent/"
+        }
+    }
+}