diff --git a/Agents/AirQualityAgent/Dockerfile b/Agents/AirQualityAgent/Dockerfile index 31672b126c1..1f18d7a5b3d 100644 --- a/Agents/AirQualityAgent/Dockerfile +++ b/Agents/AirQualityAgent/Dockerfile @@ -2,7 +2,7 @@ #================================================================================================== # Reference published Docker image for Stack-Client resources to use -FROM docker.cmclinnovations.com/stack-client:1.6.2 as stackclients +FROM ghcr.io/cambridge-cares/stack-client:1.40.1 as stackclients #------------------------------------------------------ # Base image to be reused diff --git a/Agents/AirQualityAgent/README.md b/Agents/AirQualityAgent/README.md index eaccf677d43..3135399410f 100644 --- a/Agents/AirQualityAgent/README.md +++ b/Agents/AirQualityAgent/README.md @@ -33,11 +33,7 @@ docker login ghcr.io -u ``` -### **3) Accessing CMCL docker registry** - -The agent requires building the [Stack-Clients] resource from a Docker image published at the CMCL docker registry. In case you don't have credentials for that, please email `supportcmclinnovations.com` with the subject `Docker registry access`. Further information can be found at the [CMCL Docker Registry] wiki page. - -### **4) VS Code specifics** +### **3) VS Code specifics** In order to avoid potential launching issues using the provided `tasks.json` shell commands, please ensure the `augustocdias.tasks-shell-input` plugin is installed. @@ -59,10 +55,12 @@ bash ./stack.sh remove -v After spinning up the stack, the GUI endpoints to the running containers can be accessed via Browser (i.e. adminer, blazegraph, ontop, geoserver). The endpoints and required log-in settings can be found in the [spin up the stack] readme. +Alternatively, you may copy [airqualityagent.json](stack-manager-config/inputs/config/services/airqualityagent.json) in the [stack-manager config directory](https://github.com/cambridge-cares/TheWorldAvatar/tree/main/Deploy/stacks/dynamic/stack-manager/inputs/config/services). Note that the Air Quality Agent is now exposed on localhost port 3838 instead. +   ## 1.3 Deploying the agent to the stack -This agent requires [JPS_BASE_LIB] and [Stack-Clients] to be wrapped by [py4jps]. Therefore, after installation of all required packages (incl. `py4jps >= 1.0.30`), the `StackClients` resource needs to be added to allow for access through `py4jps`. All required steps are detailed in the [py4jps] documentation. However, the following information should suffice in this context: +This agent requires [JPS_BASE_LIB] and [Stack-Clients] to be wrapped by [py4jps]. Therefore, after installation of all required packages (incl. `py4jps >= 1.0.38`), the `StackClients` resource needs to be added to allow for access through `py4jps`. All required steps are detailed in the [py4jps] documentation. However, the following information should suffice in this context: * When building the Docker images, the `StackClients` resource is copied from the published Docker image (details in the [Dockerfile]) * For testing purposes, the latest `StackClients` resource needs to be compiled and installed locally using the [py4jps] Resource Manager @@ -77,7 +75,7 @@ bash ./stack.sh build bash ./stack.sh start ``` -In case of time out issues in automatically building the StackClients resource, please try pulling the required stack-clients image first by `docker pull docker.cmclinnovations.com/stack-client:1.6.2` +In case of time out issues in automatically building the StackClients resource, please try pulling the required stack-clients image first by `docker pull ghcr.io/cambridge-cares/stack-client:1.40.1` The *debug version* will run when built and launched through the provided VS Code `launch.json` configurations: > **Build and Debug**: Build Debug Docker image (incl. pushing to [Github container registry]) and deploy as new container (incl. creation of new `.vscode/port.txt` file) @@ -135,10 +133,13 @@ Agent start-up will automatically register a recurring task to assimilate latest - GET request to update all UK-AIR stations and associated readings, and add latest data for all time series (i.e. instantiate missing stations and readings and append latest time series readings) > `/airqualityagent/update/all` - GET request to retrieve data about UK-AIR stations and create respective JSON output files (i.e. request expects all individual query parameter to be provided in a single nested JSON object with key 'query') - **please note** that this query was required to create DTVF input files previously and is now deprecated as DTVF retrieves visualisation input from PostGIS via Geoserver. This endpoint is mainly kept here for reference purposes. -> `/api/metofficeagent/retrieve/all` +> `/airqualityagent/retrieve/all` Example requests are provided in the [resources] folder, which also contain further information about allowed parameters. +## Note + +It has been observed that the UK-AIR API could be unstable, in particular for time series updates. If you receive an error message after a call to the agent, please try again later.   # 3. Agent Tests diff --git a/Agents/AirQualityAgent/agent/datainstantiation/readings.py b/Agents/AirQualityAgent/agent/datainstantiation/readings.py index 4ffb7244d0a..4248654c526 100644 --- a/Agents/AirQualityAgent/agent/datainstantiation/readings.py +++ b/Agents/AirQualityAgent/agent/datainstantiation/readings.py @@ -515,6 +515,22 @@ def retrieve_timeseries_information_from_api(ts_ids=[], days_back=7) -> dict: return infos +def repeat_http_post(url, body, headers, max_try = 3): + """ + This will send a HTTP POST request and re-try if it fails. + Currently only used for time series retrieval as it is found to be unstable. + """ + counter = 0 + while counter < max_try: + try: + r = requests.post(url=url, data=json.dumps(body), headers=headers) + # Extract time series data + return r.json() + except Exception as ex: + logger.warning(f"Error while retrieving time series data from API: {ex}") + counter = counter + 1 + time.sleep(5) + raise APIException("Error while retrieving time series data from API. Please try again later.") def retrieve_timeseries_data_from_api(crs: str = 'EPSG:4326', ts_ids=[], period='P1D', chunksize=25) -> dict: @@ -566,13 +582,7 @@ def retrieve_timeseries_data_from_api(crs: str = 'EPSG:4326', ts_ids=[], "timeseries": chunk } - try: - r = requests.post(url=url, data=json.dumps(body), headers=headers) - # Extract time series data - ts_data = r.json() - except Exception as ex: - logger.error(f"Error while retrieving time series data from API: {ex}") - raise APIException("Error while retrieving time series data from API.") from ex + ts_data = repeat_http_post(url, body, headers) df = pd.DataFrame.from_dict(ts_data, orient='index') # Remove rows without entries diff --git a/Agents/AirQualityAgent/agent/kgutils/stackclients.py b/Agents/AirQualityAgent/agent/kgutils/stackclients.py index e8909efb534..49443d15731 100644 --- a/Agents/AirQualityAgent/agent/kgutils/stackclients.py +++ b/Agents/AirQualityAgent/agent/kgutils/stackclients.py @@ -40,7 +40,6 @@ class StackClient: class OntopClient(StackClient): def __init__(self, query_endpoint=ONTOP_URL): - # Initialise OntopClient as RemoteStoreClient try: self.ontop_client = self.jpsBaseLib_view.RemoteStoreClient(query_endpoint) except Exception as ex: @@ -70,7 +69,8 @@ def upload_ontop_mapping(): f = OntopClient.stackClients_view.java.io.File(ONTOP_FILE) fp = f.toPath() # Update ONTOP mapping (requires JAVA path object) - OntopClient.stackClients_view.OntopClient().updateOBDA(fp) + ontop_client_instance = OntopClient.stackClients_view.OntopClient.getInstance() + ontop_client_instance.updateOBDA(fp) except Exception as ex: logger.error("Unable to update OBDA mapping.") raise StackException("Unable to update OBDA mapping.") from ex @@ -185,7 +185,7 @@ class GdalClient(StackClient): def __init__(self): # Initialise GdalClient with default upload/conversion settings try: - self.client = self.stackClients_view.GDALClient() + self.client = self.stackClients_view.GDALClient.getInstance() self.orgoptions = self.stackClients_view.Ogr2OgrOptions() except Exception as ex: logger.error("Unable to initialise GdalClient.") @@ -196,7 +196,7 @@ def uploadGeoJSON(self, geojson_string, database=DATABASE, table=LAYERNAME): """ Calls StackClient function with default upload settings """ - self.client.uploadVectorStringToPostGIS(database, table, geojson_string, + self.client.uploadVectorStringToPostGIS(database, "public", table, geojson_string, self.orgoptions, True) @@ -206,7 +206,7 @@ def __init__(self): # Initialise Geoserver with default settings try: - self.client = self.stackClients_view.GeoServerClient() + self.client = self.stackClients_view.GeoServerClient.getInstance() self.vectorsettings = self.stackClients_view.GeoServerVectorSettings() except Exception as ex: logger.error("Unable to initialise GeoServerClient.") @@ -225,7 +225,7 @@ def create_postgis_layer(self, geoserver_workspace=GEOSERVER_WORKSPACE, Please note: Postgis database table assumed to have same name as Geoserver layer """ - self.client.createPostGISLayer(None, geoserver_workspace, postgis_database, + self.client.createPostGISLayer(geoserver_workspace, postgis_database, "public", geoserver_layer, self.vectorsettings) diff --git a/Agents/AirQualityAgent/agent/utils/stack_configs.py b/Agents/AirQualityAgent/agent/utils/stack_configs.py index f396006cc70..ecf58ec4b07 100644 --- a/Agents/AirQualityAgent/agent/utils/stack_configs.py +++ b/Agents/AirQualityAgent/agent/utils/stack_configs.py @@ -38,7 +38,7 @@ def retrieve_settings(): pg = stackClientsView.PostGISEndpointConfig("","","","","") pg_conf = containerClient.readEndpointConfig("postgis", pg.getClass()) # Ontop - ont = stackClientsView.OntopEndpointConfig("","","","","") + ont = stackClientsView.OntopEndpointConfig("","","","") ont_conf = containerClient.readEndpointConfig("ontop", ont.getClass()) # Extract PostgreSQL/PostGIS database URL diff --git a/Agents/AirQualityAgent/app_entry_point.sh b/Agents/AirQualityAgent/app_entry_point.sh index f82bd283dd3..ce2d797a51b 100644 --- a/Agents/AirQualityAgent/app_entry_point.sh +++ b/Agents/AirQualityAgent/app_entry_point.sh @@ -1,3 +1,3 @@ #!/bin/bash # timeout set to 120min to avoid exceptions for long API/KG calls -gunicorn --bind 0.0.0.0:5000 agent.flaskapp.wsgi:app --timeout 7200 \ No newline at end of file +gunicorn --bind 0.0.0.0:5000 agent.flaskapp.wsgi:app --timeout 7200 --preload \ No newline at end of file diff --git a/Agents/AirQualityAgent/docker-compose.yml b/Agents/AirQualityAgent/docker-compose.yml index bd46714503a..1bf3ce71261 100644 --- a/Agents/AirQualityAgent/docker-compose.yml +++ b/Agents/AirQualityAgent/docker-compose.yml @@ -2,7 +2,7 @@ version: "3.8" services: airquality_agent: - image: ghcr.io/cambridge-cares/airquality_agent:1.0.0 + image: ghcr.io/cambridge-cares/airquality_agent:1.1.0-SNAPSHOT environment: - STACK_NAME=${STACK_NAME} # Target Blazegraph namespace diff --git a/Agents/AirQualityAgent/setup.py b/Agents/AirQualityAgent/setup.py index 1af4cbf10ec..5efcafd4d2f 100644 --- a/Agents/AirQualityAgent/setup.py +++ b/Agents/AirQualityAgent/setup.py @@ -16,9 +16,11 @@ install_requires= [ 'APScheduler==3.9.1', 'Flask==2.2.2', + 'werkzeug==2.2.2', 'JayDeBeApi==1.2.3', 'pandas~=1.4', - 'py4jps==1.0.34', + 'numpy~=1.24', + 'py4jps==1.0.38', 'requests==2.28.1', ] ) \ No newline at end of file diff --git a/Agents/AirQualityAgent/stack-manager-config/inputs/config/services/airqualityagent.json b/Agents/AirQualityAgent/stack-manager-config/inputs/config/services/airqualityagent.json new file mode 100644 index 00000000000..af5934b8035 --- /dev/null +++ b/Agents/AirQualityAgent/stack-manager-config/inputs/config/services/airqualityagent.json @@ -0,0 +1,59 @@ +{ + "ServiceSpec": { + "Name": "air-quality-agent", + "TaskTemplate": { + "ContainerSpec": { + "Image": "ghcr.io/cambridge-cares/airquality_agent:1.1.0-SNAPSHOT", + "Env": [ + "STACK_NAME=${STACK_NAME}", + "NAMESPACE=airquality", + "LAYERNAME=airquality", + "DATABASE=postgres", + "GEOSERVER_WORKSPACE=stations", + "ONTOP_FILE=/app/resources/ontop.obda" + ], + "Mounts": [ + { + "Type": "bind", + "Source": "../../../../../../Agents/AirQualityAgent/output", + "Target": "/app/output" + }, + { + "Type": "volume", + "Source": "logs", + "Target": "/root/.jps" + } + ], + "Configs": [ + { + "ConfigName": "blazegraph" + }, + { + "ConfigName": "geoserver" + }, + { + "ConfigName": "ontop" + }, + { + "ConfigName": "postgis" + } + ], + "Secrets": [ + { + "SecretName": "postgis_password" + }, + { + "SecretName": "geoserver_password" + } + + ] + } + } + }, + "endpoints": { + "ui": { + "url": "http://localhost:5000/airqualityagent/", + "externalPath": "/airqualityagent/" + } + } +} \ No newline at end of file