Making NGSI-LDF work on top of real world context broker

README.md

@@ -35,13 +35,15 @@ The root directory contains a configuration file (`config.toml`) that contains t
 ```toml
 [ngsi]
   # Location of the NGSI-LD endpoint
-  host = "http://localhost:3000"
+  host = "http://localhost:9090/ngsi-ld/v1"
 
 [api]
   # Base URI of the generated fragments
   host = "http://localhost:3001"
-  # number of observations to include in the /latest fragments
-  lastN = 100
+  # time range (last number of minutes) to include in the /latest fragments (lastN is not supported with Scorpio Context Broker)
+  lastNumberOfMinutes = 100
+  # Use NGSI metadata model or simplified keyValues
+  keyValues = true
 
 [data]
   # NGSI-LD exclusively uses relative property URIs
@@ -51,6 +53,9 @@ The root directory contains a configuration file (`config.toml`) that contains t
   metrics = ["NO2", "O3", "PM10", "PM1", "PM25"]
 ```
 
+The publisher can configure whether to use the simplified NGSI-LD output (keyValues) or not. 
+When using RDFS vocabularies, it's necessary to use keyValues to obtain semantically correct triples.
+
 ## Fragmentations
 
 The fragmentation strategy ultimately determines the usability of the published Linked Data Fragments. Making them too granular eliminates the cacheability of the data, making them too course makes the data harder to ingest for consumers. Keeping this in mind, we currently support three common geospatial bucketing strategies and a flexible temporal fragmentation strategy.
@@ -100,10 +105,11 @@ Temporal fragmentations are also used to aggregate data, in which case we can be
 
 The raw data interfaces adhere to any of these templates:
 
-* **Slippy**: `/{z}/{x}/{y}{?page}`
-* **Geohash**: `/geohash/{hash}{?page}`
-* **H3**: `/h3/{index}{?page}`
+* **Slippy**: `/{z}/{x}/{y}{?type,page}`
+* **Geohash**: `/geohash/{hash}{?type,page}`
+* **H3**: `/h3/{index}{?type,page}`
 
+Where `type` corresponds with the type of instances that need to be queried. This needs to be URI encoded, e.g. https%3A%2F%2Fdata.vlaanderen.be%2Fns%2Fgebouw%23Gebouw
 Where `page` corresponds to a XSD DateTime string in UTC. If no page starts at the specified time, the page containing that time is returned instead. All other parameters are defined in the Geospatial Fragmentations section.
 
 The `@graph` element of the returned data contains the temporal representation of all entities in the specified area, following the NGSI-LD representation for 'Query Temporal Evolution of Entities' operations, with the exception that individuals don't (necessarily) contain their own contexts. This knowledge graph can be fed into NGSI-lD compatible clients. 
@@ -114,9 +120,9 @@ General-purpose JSON-LD clients must be wary of the differences between JSON-LD
 
 The latest data interfaces adhere to any of these templates:
 
-- **Slippy**: `/{z}/{x}/{y}/latest`
-- **Geohash**: `/geohash/{hash}/latest`
-- **H3**: `/h3/{index}/latest`
+- **Slippy**: `/{z}/{x}/{y}/latest{?type}`
+- **Geohash**: `/geohash/{hash}/latest{?type}`
+- **H3**: `/h3/{index}/latest{?type}`
 
 This interface returns raw data as well, and as such shares most of its properties with the raw data one. The difference is that this interface returns tiles that contain a fixed amount of observations -- instead of temporally fragmented pages. This is interface is more useful for application that are interested in real-time  data, as they can just periodically poll for new data. The raw data interface on the other hand is mostly useful for application that are looking for historical data.
 
@@ -166,33 +172,45 @@ The server's responses have 4 kinds of equally important headers:
 
 #### Search Template
 
-Each fragment defines its own URI structure so that clients that know which other fragments they're looking for can simply fill in the template and arrive at their destination. This kind of hypermedia control is defined using the [hydra](https://www.hydra-cg.com/spec/latest/core/) vocabulary. For example, this is a possible search template for the aggregate data:
+Each fragment defines its own URI structure so that clients that know which other fragments they're looking for can simply fill in the template and arrive at their destination. This kind of hypermedia control is defined using the [hydra](https://www.hydra-cg.com/spec/latest/core/) vocabulary. For example, this is a search template for the raw data:
 
 ```json
 "hydra:search": {
   "@type": "hydra:IriTemplate",
-  "hydra:template": "http://localhost:3001/geohash/{hash}/summary{?page,period}",
+  "hydra:template": "http://localhost:3001/{z}/{x}/{y}{?type,page}",
   "hydra:variableRepresentation": "hydra:BasicRepresentation",
   "hydra:mapping": [
-    {
-      "@type": "hydra:IriTemplateMapping",
-      "hydra:variable": "hash",
-      "hydra:property": "tiles:geohash",
-      "hydra:required": true
-    },
-    {
-      "@type": "hydra:IriTemplateMapping",
-      "hydra:variable": "page",
-      "hydra:property": "schema:startDate",
-      "hydra:required": false
-    },
-    {
-      "@type": "hydra:IriTemplateMapping",
-      "hydra:variable": "period",
-      "hydra:property": "cot:hasAggregationPeriod",
-      "hydra:required": false
-    }
-  ]
+       {
+           @type: "hydra:IriTemplateMapping",
+           hydra:variable: "z",
+           hydra:property: "tiles:zoom",
+           hydra:required: true
+       },
+       {
+           @type: "hydra:IriTemplateMapping",
+           hydra:variable: "x",
+           hydra:property: "tiles:longitudeTile",
+           hydra:required: true
+       },
+       {
+           @type: "hydra:IriTemplateMapping",
+           hydra:variable: "y",
+           hydra:property: "tiles:latitudeTile",
+           hydra:required: true
+       },
+       {
+           @type: "hydra:IriTemplateMapping",
+           hydra:variable: "type",
+           hydra:property: "rdf:type",
+           hydra:required: true
+       },
+       {
+           @type: "hydra:IriTemplateMapping",
+           hydra:variable: "page",
+           hydra:property: "schema:startDate",
+           hydra:required: false
+       }
+   ]
 }
 ```
 
@@ -205,21 +223,19 @@ Individual fragments are also linked together, so that data consumers can traver
 ![tree](img/tree.svg)
 
 
-
 In essence:
 
-* Raw and summary data pages contain links to the next and previous pages (respectively using `tree:GreaterThanRelation` and `tree:LesserThanRelation` links).
+* Raw ~~and summary~~ data pages contain links to the next and previous pages (respectively using `tree:GreaterThanRelation` and `tree:LessThanRelation` links).
 * The active raw data page, that is the one that contains observations happening right now, is linked with the latest data fragment with `tree:AlternativeViewRelation` links.
-* The summary data pages refer to the raw data pages that were used to compute the aggregate values with `tree:DerivedFromRelation` links.
 
 ## Equivalent NGSI-LD Requests
 
 Incoming raw data or latest data requests get translated to NGSI-LD requests, the following table contains some examples:
 
 |        | NGSI-LDF                                      | NGSI-LD                                                      |
 | ------ | --------------------------------------------- | ------------------------------------------------------------ |
-| Raw    | `/14/8393/5467?page=2019-11-06T15:00:00.000Z` | `/temporal/entities?georel=within&geometry=Polygon&coordinates=[[[4.41650390625,51.248163159055906],[4.41650390625,51.2344073516346],[4.4384765625,51.2344073516346],[4.4384765625,51.2344073516346]]]&timerel=between&time=2019-11-06T15:00:00.000Z&endTime=2019-11-06T16:00:00.000Z` |
-| Latest | `/14/8393/5467/latest`                        | `/temporal/entities?georel=within&geometry=Polygon&coordinates=[[[4.41650390625,51.248163159055906],[4.41650390625,51.2344073516346],[4.4384765625,51.2344073516346],[4.4384765625,51.2344073516346]]]&timerel=before&time=2019-11-06T16:04:43.640Z&lastN=100` |
+| Raw    | `/14/8393/5467?type=https%3A%2F%2Fdata.vlaanderen.be%2Fns%2Fgebouw%23Gebouw&page=2019-11-06T15:00:00.000Z` | `/temporal/entities?type=https%3A%2F%2Fdata.vlaanderen.be%2Fns%2Fgebouw%23Gebouw&georel=within&geometry=Polygon&coordinates=[[[4.41650390625,51.248163159055906],[4.41650390625,51.2344073516346],[4.4384765625,51.2344073516346],[4.4384765625,51.2344073516346]]]&timerel=between&time=2019-11-06T15:00:00.000Z&endTime=2019-11-06T16:00:00.000Z&timeproperty=modifiedAt&options=sysAttrs` |
+| Latest | `/14/8393/5467/latest?type=https%3A%2F%2Fdata.vlaanderen.be%2Fns%2Fgebouw%23Gebouw`                        | `/temporal/entities?type=https%3A%2F%2Fdata.vlaanderen.be%2Fns%2Fgebouw%23Gebouw&georel=within&geometry=Polygon&coordinates=[[[4.41650390625,51.248163159055906],[4.41650390625,51.2344073516346],[4.4384765625,51.2344073516346],[4.4384765625,51.2344073516346]]]&timerel=between&time=2019-11-05T16:04:43.640Z&endTime=2019-11-06T16:04:43.640Z&timeproperty=modifiedAt&options=sysAttrs` |
 
 
 

config.toml

@@ -1,12 +1,14 @@
 [ngsi]
   # Location of the NGSI-LD endpoint
-  host = "http://localhost:3000"
+  host = "http://localhost:9090/ngsi-ld/v1"
 
 [api]
   # Base URI of the generated fragments
   host = "http://localhost:3001"
-  # number of observations to include in the /latest fragments
-  lastN = 100
+  # time range (last number of minutes) to include in the /latest fragments
+  lastNumberOfMinutes = 100
+  # Use NGSI metadata model or simplified keyValues
+  keyValues = true
 
 [data]
   # NGSI-LD exclusively uses relative property URIs

package.json

@@ -19,6 +19,7 @@
     "debug": "^2.2.0",
     "express": "^4.17.1",
     "h3-js": "^3.6.1",
+    "jsonld-context-parser": "^2.1.1",
     "md5": "^2.2.1",
     "ngeohash": "^0.6.3",
     "node-fetch": "^2.6.0",

src/config/config.ts

@@ -5,7 +5,8 @@ interface IConfig {
     sourceURI: string;
     targetURI: string;
     metrics: string[];
-    lastN: number;
+    lastNumberOfMinutes: number;
+    keyValues: boolean;
 }
 
 let config: IConfig;
@@ -16,7 +17,8 @@ export function getConfig(): IConfig {
             sourceURI: data.ngsi.host,
             targetURI: data.api.host,
             metrics: data.data.metrics,
-            lastN: data.api.lastN,
+            lastNumberOfMinutes: data.api.lastNumberOfMinutes,
+            keyValues: data.api.keyValues
         };
     }
 

src/controllers/dataController.ts

@@ -6,12 +6,12 @@ import GeohashFragmenter from "../fragmenters/geohash";
 import H3Fragmenter from "../fragmenters/h3";
 import SlippyFragmenter from "../fragmenters/slippy";
 import TimeFragmenter from "../fragmenters/time";
+import {convertResponseToEventstream, expandVocabulary, extractVocabulary, simplifyGraph} from "../utils/Util";
 
-// tslint:disable: no-string-literal
-
-function wrapPage(
+async function wrapPage(
     req: Request, // the original request
     data: object, // the converted data
+    type: string, // the type of objects
     timeFragmenter: TimeFragmenter, // temporal fragmentation strategy
     geoFragmenter: GeoFragmenter, // geospatial fragmentation strategy
 ) {
@@ -25,17 +25,17 @@ function wrapPage(
     const previousTime = timeFragmenter.getPreviousTime(fromTime);
 
     // adapt/use a new json-ld context
-    const vocabulary = extractVocubalary(data);
-    expandVocabulary(vocabulary);
-    simplifyGraph(vocabulary, data);
+    let vocabulary = extractVocabulary(data);
+    vocabulary = await expandVocabulary(vocabulary);
+    // simplifyGraph(expandedVocabulary, data);
 
     const config = getConfig();
 
     // add links to previous/next pages
     const children = [{
-        "@type": "tree:LesserThanRelation",
-        "tree:node": geoFragmenter.getDataFragmentURI(config.targetURI, focus, precision, previousTime),
-        "sh:path": "ngsi-ld:observedAt",
+        "@type": "tree:LessThanRelation",
+        "tree:node": geoFragmenter.getDataFragmentURI(config.targetURI, type, focus, precision, previousTime),
+        "tree:path": "ngsi-ld:modifiedAt",
         "tree:value": {
             "schema:startDate": previousTime.toISOString(),
             "schema:endDate": fromTime.toISOString(),
@@ -45,8 +45,8 @@ function wrapPage(
     if (new Date() > nextTime) {
         children.push({
             "@type": "tree:GreaterThanRelation",
-            "tree:node": geoFragmenter.getDataFragmentURI(config.targetURI, focus, precision, nextTime),
-            "sh:path": "ngsi-ld:observedAt",
+            "tree:node": geoFragmenter.getDataFragmentURI(config.targetURI, type, focus, precision, nextTime),
+            "tree:path": "ngsi-ld:modifiedAt",
             "tree:value": {
                 "schema:startDate": nextTime.toISOString(),
                 "schema:endDate": timeFragmenter.getNextTime(nextTime).toISOString(),
@@ -55,8 +55,8 @@ function wrapPage(
     } else {
         children.push({
             "@type": "tree:AlternateViewRelation",
-            "tree:node": geoFragmenter.getLatestFragmentURI(config.targetURI, focus, precision),
-            "sh:path": "ngsi-ld:observedAt",
+            "tree:node": geoFragmenter.getLatestFragmentURI(config.targetURI, type, focus, precision),
+            "tree:path": "ngsi-ld:modifiedAt",
             "tree:value": {
                 "schema:startDate": undefined,
                 "schema:endDate": new Date().toISOString(),
@@ -67,21 +67,21 @@ function wrapPage(
     // build the fragment
     const result = {
         "@context": vocabulary,
-        "@id": geoFragmenter.getDataFragmentURI(config.targetURI, focus, precision, fromTime),
+        "@id": geoFragmenter.getDataFragmentURI(config.targetURI, type, focus, precision, fromTime),
         "@type": "tree:Node",
         ...geoFragmenter.getMetaData(focus, precision),
         "tree:relation": children,
-        "sh:path": "ngsi-ld:observedAt",
+        "tree:path": "ngsi-ld:modifiedAt",
         "tree:value": {
             "schema:startDate": fromTime.toISOString(),
             "schema:endDate": nextTime.toISOString(),
         },
         "dcterms:isPartOf": {
-            "@id": config.targetURI,
-            "@type": "hydra:Collection",
+            "@id": `${config.targetURI}/${encodeURIComponent(type)}`,
+            "@type": "tree:Collection",
             "hydra:search": geoFragmenter.getDataSearchTemplate(config.targetURI),
         },
-        "@graph": data,
+        "@included": data,
     };
 
     return result;
@@ -114,19 +114,29 @@ async function getPage(
     const focus = geoFragmenter.getFocusPoint(req);
     const bbox = [geoFragmenter.getBBox(focus, precision).map((location) => [location.longitude, location.latitude])];
 
+    // type for NGSI-LD
+    if (!req.query.type) {
+        res.status(404).send("Provide \"type\" query parameter");
+        return;
+    }
+    const type = decodeURIComponent(req.query.type.toString());
+
     const config = getConfig();
 
-    const uri = `${config.sourceURI}/temporal/entities?georel=within&geometry=Polygon&`
+    const uri = `${config.sourceURI}/temporal/entities?type=${encodeURIComponent(type)}&georel=within&geometry=Polygon&`
         + `coordinates=${JSON.stringify(bbox)}&timerel=between`
-        + `&time=${fromTime.toISOString()}&endTime=${toTime.toISOString()}`;
+        + `&time=${fromTime.toISOString()}&endTime=${toTime.toISOString()}`
+        + `&timeproperty=modifiedAt&options=sysAttrs`;
     const response = await fetch(uri);
 
     // remember when the response arrived
     const dataTime = new Date();
-    const data = await response.json();
+    const responseJson = await response.json();
+
+    const data = await convertResponseToEventstream(responseJson, type, fromTime, toTime, bbox);
 
     // add metadata to the resulting data
-    const pagedData = wrapPage(req, data, timeFragmenter, geoFragmenter);
+    const pagedData = await wrapPage(req, data, type, timeFragmenter, geoFragmenter);
 
     // fragment is presumed stable if the 'toTime' lies in the past
     addHeaders(res, toTime < dataTime);
@@ -160,62 +170,4 @@ export async function getGeohashPage(req, res) {
 export async function getH3Page(req, res) {
     const geoFragmenter = new H3Fragmenter();
     await getPage(req, res, new TimeFragmenter(TimeFragmenter.HOUR), geoFragmenter);
-}
-
-/*
- * Placeholders
- * There are probably libraries that automate the mutating of contexts
- * Ideally these functions create and use a new vocabulary combining the original data and the derived view
- */
-
-function extractVocubalary(data) {
-    // fixme; simple placeholder
-    if (data && data.length) {
-        return data[0]["@context"];
-    } else {
-        return {};
-    }
-}
-
-function expandVocabulary(vocabulary) {
-    let targetContext;
-    if (vocabulary && vocabulary.length) {
-        targetContext = vocabulary[0];
-    } else {
-        targetContext = vocabulary;
-    }
-
-    targetContext["xsd"] = "http://www.w3.org/2001/XMLSchema#";
-    targetContext["schema"] = "http://schema.org/";
-    targetContext["schema:startDate"] = {
-        "@type": "xsd:dateTime",
-    };
-    targetContext["schema:endDate"] = {
-        "@type": "xsd:dateTime",
-    };
-    targetContext["dcterms"] = "http://purl.org/dc/terms/";
-    targetContext["tree"] = "https://w3id.org/tree/terms#";
-    targetContext["tree:node"] = {
-        "@type": "@id",
-    };
-    targetContext["tiles"] = "https://w3id.org/tree/terms#";
-    targetContext["hydra"] = "http://www.w3.org/ns/hydra/core#";
-    targetContext["hydra:variableRepresentation"] = {
-        "@type": "@id",
-    };
-    targetContext["hydra:property"] = {
-        "@type": "@id",
-    };
-    targetContext["sh"] = "https://www.w3.org/ns/shacl#";
-    targetContext["sh:path"] = {
-        "@type": "@id",
-    };
-    targetContext["ngsi-ld"] = "https://uri.etsi.org/ngsi-ld/";
-}
-
-function simplifyGraph(vocabulary, graph) {
-    // fixme; simple placeholder
-    for (const entity of graph) {
-        delete entity["@context"];
-    }
-}
+}