diff --git a/README.md b/README.md index 54d5eab..5e10861 100644 --- a/README.md +++ b/README.md @@ -35,13 +35,15 @@ The root directory contains a configuration file (`config.toml`) that contains t ```toml [ngsi] # Location of the NGSI-LD endpoint - host = "http://localhost:3000" + host = "http://localhost:9090/ngsi-ld/v1" [api] # Base URI of the generated fragments host = "http://localhost:3001" - # number of observations to include in the /latest fragments - lastN = 100 + # time range (last number of minutes) to include in the /latest fragments (lastN is not supported with Scorpio Context Broker) + lastNumberOfMinutes = 100 + # Use NGSI metadata model or simplified keyValues + keyValues = true [data] # NGSI-LD exclusively uses relative property URIs @@ -51,6 +53,9 @@ The root directory contains a configuration file (`config.toml`) that contains t metrics = ["NO2", "O3", "PM10", "PM1", "PM25"] ``` +The publisher can configure whether to use the simplified NGSI-LD output (keyValues) or not. +When using RDFS vocabularies, it's necessary to use keyValues to obtain semantically correct triples. + ## Fragmentations The fragmentation strategy ultimately determines the usability of the published Linked Data Fragments. Making them too granular eliminates the cacheability of the data, making them too course makes the data harder to ingest for consumers. Keeping this in mind, we currently support three common geospatial bucketing strategies and a flexible temporal fragmentation strategy. @@ -100,10 +105,11 @@ Temporal fragmentations are also used to aggregate data, in which case we can be The raw data interfaces adhere to any of these templates: -* **Slippy**: `/{z}/{x}/{y}{?page}` -* **Geohash**: `/geohash/{hash}{?page}` -* **H3**: `/h3/{index}{?page}` +* **Slippy**: `/{z}/{x}/{y}{?type,page}` +* **Geohash**: `/geohash/{hash}{?type,page}` +* **H3**: `/h3/{index}{?type,page}` +Where `type` corresponds with the type of instances that need to be queried. This needs to be URI encoded, e.g. https%3A%2F%2Fdata.vlaanderen.be%2Fns%2Fgebouw%23Gebouw Where `page` corresponds to a XSD DateTime string in UTC. If no page starts at the specified time, the page containing that time is returned instead. All other parameters are defined in the Geospatial Fragmentations section. The `@graph` element of the returned data contains the temporal representation of all entities in the specified area, following the NGSI-LD representation for 'Query Temporal Evolution of Entities' operations, with the exception that individuals don't (necessarily) contain their own contexts. This knowledge graph can be fed into NGSI-lD compatible clients. @@ -114,9 +120,9 @@ General-purpose JSON-LD clients must be wary of the differences between JSON-LD The latest data interfaces adhere to any of these templates: -- **Slippy**: `/{z}/{x}/{y}/latest` -- **Geohash**: `/geohash/{hash}/latest` -- **H3**: `/h3/{index}/latest` +- **Slippy**: `/{z}/{x}/{y}/latest{?type}` +- **Geohash**: `/geohash/{hash}/latest{?type}` +- **H3**: `/h3/{index}/latest{?type}` This interface returns raw data as well, and as such shares most of its properties with the raw data one. The difference is that this interface returns tiles that contain a fixed amount of observations -- instead of temporally fragmented pages. This is interface is more useful for application that are interested in real-time data, as they can just periodically poll for new data. The raw data interface on the other hand is mostly useful for application that are looking for historical data. @@ -166,33 +172,45 @@ The server's responses have 4 kinds of equally important headers: #### Search Template -Each fragment defines its own URI structure so that clients that know which other fragments they're looking for can simply fill in the template and arrive at their destination. This kind of hypermedia control is defined using the [hydra](https://www.hydra-cg.com/spec/latest/core/) vocabulary. For example, this is a possible search template for the aggregate data: +Each fragment defines its own URI structure so that clients that know which other fragments they're looking for can simply fill in the template and arrive at their destination. This kind of hypermedia control is defined using the [hydra](https://www.hydra-cg.com/spec/latest/core/) vocabulary. For example, this is a search template for the raw data: ```json "hydra:search": { "@type": "hydra:IriTemplate", - "hydra:template": "http://localhost:3001/geohash/{hash}/summary{?page,period}", + "hydra:template": "http://localhost:3001/{z}/{x}/{y}{?type,page}", "hydra:variableRepresentation": "hydra:BasicRepresentation", "hydra:mapping": [ - { - "@type": "hydra:IriTemplateMapping", - "hydra:variable": "hash", - "hydra:property": "tiles:geohash", - "hydra:required": true - }, - { - "@type": "hydra:IriTemplateMapping", - "hydra:variable": "page", - "hydra:property": "schema:startDate", - "hydra:required": false - }, - { - "@type": "hydra:IriTemplateMapping", - "hydra:variable": "period", - "hydra:property": "cot:hasAggregationPeriod", - "hydra:required": false - } - ] + { + @type: "hydra:IriTemplateMapping", + hydra:variable: "z", + hydra:property: "tiles:zoom", + hydra:required: true + }, + { + @type: "hydra:IriTemplateMapping", + hydra:variable: "x", + hydra:property: "tiles:longitudeTile", + hydra:required: true + }, + { + @type: "hydra:IriTemplateMapping", + hydra:variable: "y", + hydra:property: "tiles:latitudeTile", + hydra:required: true + }, + { + @type: "hydra:IriTemplateMapping", + hydra:variable: "type", + hydra:property: "rdf:type", + hydra:required: true + }, + { + @type: "hydra:IriTemplateMapping", + hydra:variable: "page", + hydra:property: "schema:startDate", + hydra:required: false + } + ] } ``` @@ -205,12 +223,10 @@ Individual fragments are also linked together, so that data consumers can traver ![tree](img/tree.svg) - In essence: -* Raw and summary data pages contain links to the next and previous pages (respectively using `tree:GreaterThanRelation` and `tree:LesserThanRelation` links). +* Raw ~~and summary~~ data pages contain links to the next and previous pages (respectively using `tree:GreaterThanRelation` and `tree:LessThanRelation` links). * The active raw data page, that is the one that contains observations happening right now, is linked with the latest data fragment with `tree:AlternativeViewRelation` links. -* The summary data pages refer to the raw data pages that were used to compute the aggregate values with `tree:DerivedFromRelation` links. ## Equivalent NGSI-LD Requests @@ -218,8 +234,8 @@ Incoming raw data or latest data requests get translated to NGSI-LD requests, th | | NGSI-LDF | NGSI-LD | | ------ | --------------------------------------------- | ------------------------------------------------------------ | -| Raw | `/14/8393/5467?page=2019-11-06T15:00:00.000Z` | `/temporal/entities?georel=within&geometry=Polygon&coordinates=[[[4.41650390625,51.248163159055906],[4.41650390625,51.2344073516346],[4.4384765625,51.2344073516346],[4.4384765625,51.2344073516346]]]&timerel=between&time=2019-11-06T15:00:00.000Z&endTime=2019-11-06T16:00:00.000Z` | -| Latest | `/14/8393/5467/latest` | `/temporal/entities?georel=within&geometry=Polygon&coordinates=[[[4.41650390625,51.248163159055906],[4.41650390625,51.2344073516346],[4.4384765625,51.2344073516346],[4.4384765625,51.2344073516346]]]&timerel=before&time=2019-11-06T16:04:43.640Z&lastN=100` | +| Raw | `/14/8393/5467?type=https%3A%2F%2Fdata.vlaanderen.be%2Fns%2Fgebouw%23Gebouw&page=2019-11-06T15:00:00.000Z` | `/temporal/entities?type=https%3A%2F%2Fdata.vlaanderen.be%2Fns%2Fgebouw%23Gebouw&georel=within&geometry=Polygon&coordinates=[[[4.41650390625,51.248163159055906],[4.41650390625,51.2344073516346],[4.4384765625,51.2344073516346],[4.4384765625,51.2344073516346]]]&timerel=between&time=2019-11-06T15:00:00.000Z&endTime=2019-11-06T16:00:00.000Z&timeproperty=modifiedAt&options=sysAttrs` | +| Latest | `/14/8393/5467/latest?type=https%3A%2F%2Fdata.vlaanderen.be%2Fns%2Fgebouw%23Gebouw` | `/temporal/entities?type=https%3A%2F%2Fdata.vlaanderen.be%2Fns%2Fgebouw%23Gebouw&georel=within&geometry=Polygon&coordinates=[[[4.41650390625,51.248163159055906],[4.41650390625,51.2344073516346],[4.4384765625,51.2344073516346],[4.4384765625,51.2344073516346]]]&timerel=between&time=2019-11-05T16:04:43.640Z&endTime=2019-11-06T16:04:43.640Z&timeproperty=modifiedAt&options=sysAttrs` | diff --git a/config.toml b/config.toml index 8980ddb..da5a2a6 100644 --- a/config.toml +++ b/config.toml @@ -1,12 +1,14 @@ [ngsi] # Location of the NGSI-LD endpoint - host = "http://localhost:3000" + host = "http://localhost:9090/ngsi-ld/v1" [api] # Base URI of the generated fragments host = "http://localhost:3001" - # number of observations to include in the /latest fragments - lastN = 100 + # time range (last number of minutes) to include in the /latest fragments + lastNumberOfMinutes = 100 + # Use NGSI metadata model or simplified keyValues + keyValues = true [data] # NGSI-LD exclusively uses relative property URIs diff --git a/package.json b/package.json index 6a356c2..2bbdf89 100644 --- a/package.json +++ b/package.json @@ -19,6 +19,7 @@ "debug": "^2.2.0", "express": "^4.17.1", "h3-js": "^3.6.1", + "jsonld-context-parser": "^2.1.1", "md5": "^2.2.1", "ngeohash": "^0.6.3", "node-fetch": "^2.6.0", diff --git a/src/config/config.ts b/src/config/config.ts index 5bab847..550c017 100644 --- a/src/config/config.ts +++ b/src/config/config.ts @@ -5,7 +5,8 @@ interface IConfig { sourceURI: string; targetURI: string; metrics: string[]; - lastN: number; + lastNumberOfMinutes: number; + keyValues: boolean; } let config: IConfig; @@ -16,7 +17,8 @@ export function getConfig(): IConfig { sourceURI: data.ngsi.host, targetURI: data.api.host, metrics: data.data.metrics, - lastN: data.api.lastN, + lastNumberOfMinutes: data.api.lastNumberOfMinutes, + keyValues: data.api.keyValues }; } diff --git a/src/controllers/dataController.ts b/src/controllers/dataController.ts index ab1ca33..265e958 100644 --- a/src/controllers/dataController.ts +++ b/src/controllers/dataController.ts @@ -6,12 +6,12 @@ import GeohashFragmenter from "../fragmenters/geohash"; import H3Fragmenter from "../fragmenters/h3"; import SlippyFragmenter from "../fragmenters/slippy"; import TimeFragmenter from "../fragmenters/time"; +import {convertResponseToEventstream, expandVocabulary, extractVocabulary, simplifyGraph} from "../utils/Util"; -// tslint:disable: no-string-literal - -function wrapPage( +async function wrapPage( req: Request, // the original request data: object, // the converted data + type: string, // the type of objects timeFragmenter: TimeFragmenter, // temporal fragmentation strategy geoFragmenter: GeoFragmenter, // geospatial fragmentation strategy ) { @@ -25,17 +25,17 @@ function wrapPage( const previousTime = timeFragmenter.getPreviousTime(fromTime); // adapt/use a new json-ld context - const vocabulary = extractVocubalary(data); - expandVocabulary(vocabulary); - simplifyGraph(vocabulary, data); + let vocabulary = extractVocabulary(data); + vocabulary = await expandVocabulary(vocabulary); + // simplifyGraph(expandedVocabulary, data); const config = getConfig(); // add links to previous/next pages const children = [{ - "@type": "tree:LesserThanRelation", - "tree:node": geoFragmenter.getDataFragmentURI(config.targetURI, focus, precision, previousTime), - "sh:path": "ngsi-ld:observedAt", + "@type": "tree:LessThanRelation", + "tree:node": geoFragmenter.getDataFragmentURI(config.targetURI, type, focus, precision, previousTime), + "tree:path": "ngsi-ld:modifiedAt", "tree:value": { "schema:startDate": previousTime.toISOString(), "schema:endDate": fromTime.toISOString(), @@ -45,8 +45,8 @@ function wrapPage( if (new Date() > nextTime) { children.push({ "@type": "tree:GreaterThanRelation", - "tree:node": geoFragmenter.getDataFragmentURI(config.targetURI, focus, precision, nextTime), - "sh:path": "ngsi-ld:observedAt", + "tree:node": geoFragmenter.getDataFragmentURI(config.targetURI, type, focus, precision, nextTime), + "tree:path": "ngsi-ld:modifiedAt", "tree:value": { "schema:startDate": nextTime.toISOString(), "schema:endDate": timeFragmenter.getNextTime(nextTime).toISOString(), @@ -55,8 +55,8 @@ function wrapPage( } else { children.push({ "@type": "tree:AlternateViewRelation", - "tree:node": geoFragmenter.getLatestFragmentURI(config.targetURI, focus, precision), - "sh:path": "ngsi-ld:observedAt", + "tree:node": geoFragmenter.getLatestFragmentURI(config.targetURI, type, focus, precision), + "tree:path": "ngsi-ld:modifiedAt", "tree:value": { "schema:startDate": undefined, "schema:endDate": new Date().toISOString(), @@ -67,21 +67,21 @@ function wrapPage( // build the fragment const result = { "@context": vocabulary, - "@id": geoFragmenter.getDataFragmentURI(config.targetURI, focus, precision, fromTime), + "@id": geoFragmenter.getDataFragmentURI(config.targetURI, type, focus, precision, fromTime), "@type": "tree:Node", ...geoFragmenter.getMetaData(focus, precision), "tree:relation": children, - "sh:path": "ngsi-ld:observedAt", + "tree:path": "ngsi-ld:modifiedAt", "tree:value": { "schema:startDate": fromTime.toISOString(), "schema:endDate": nextTime.toISOString(), }, "dcterms:isPartOf": { - "@id": config.targetURI, - "@type": "hydra:Collection", + "@id": `${config.targetURI}/${encodeURIComponent(type)}`, + "@type": "tree:Collection", "hydra:search": geoFragmenter.getDataSearchTemplate(config.targetURI), }, - "@graph": data, + "@included": data, }; return result; @@ -114,19 +114,29 @@ async function getPage( const focus = geoFragmenter.getFocusPoint(req); const bbox = [geoFragmenter.getBBox(focus, precision).map((location) => [location.longitude, location.latitude])]; + // type for NGSI-LD + if (!req.query.type) { + res.status(404).send("Provide \"type\" query parameter"); + return; + } + const type = decodeURIComponent(req.query.type.toString()); + const config = getConfig(); - const uri = `${config.sourceURI}/temporal/entities?georel=within&geometry=Polygon&` + const uri = `${config.sourceURI}/temporal/entities?type=${encodeURIComponent(type)}&georel=within&geometry=Polygon&` + `coordinates=${JSON.stringify(bbox)}&timerel=between` - + `&time=${fromTime.toISOString()}&endTime=${toTime.toISOString()}`; + + `&time=${fromTime.toISOString()}&endTime=${toTime.toISOString()}` + + `&timeproperty=modifiedAt&options=sysAttrs`; const response = await fetch(uri); // remember when the response arrived const dataTime = new Date(); - const data = await response.json(); + const responseJson = await response.json(); + + const data = await convertResponseToEventstream(responseJson, type, fromTime, toTime, bbox); // add metadata to the resulting data - const pagedData = wrapPage(req, data, timeFragmenter, geoFragmenter); + const pagedData = await wrapPage(req, data, type, timeFragmenter, geoFragmenter); // fragment is presumed stable if the 'toTime' lies in the past addHeaders(res, toTime < dataTime); @@ -160,62 +170,4 @@ export async function getGeohashPage(req, res) { export async function getH3Page(req, res) { const geoFragmenter = new H3Fragmenter(); await getPage(req, res, new TimeFragmenter(TimeFragmenter.HOUR), geoFragmenter); -} - -/* - * Placeholders - * There are probably libraries that automate the mutating of contexts - * Ideally these functions create and use a new vocabulary combining the original data and the derived view - */ - -function extractVocubalary(data) { - // fixme; simple placeholder - if (data && data.length) { - return data[0]["@context"]; - } else { - return {}; - } -} - -function expandVocabulary(vocabulary) { - let targetContext; - if (vocabulary && vocabulary.length) { - targetContext = vocabulary[0]; - } else { - targetContext = vocabulary; - } - - targetContext["xsd"] = "http://www.w3.org/2001/XMLSchema#"; - targetContext["schema"] = "http://schema.org/"; - targetContext["schema:startDate"] = { - "@type": "xsd:dateTime", - }; - targetContext["schema:endDate"] = { - "@type": "xsd:dateTime", - }; - targetContext["dcterms"] = "http://purl.org/dc/terms/"; - targetContext["tree"] = "https://w3id.org/tree/terms#"; - targetContext["tree:node"] = { - "@type": "@id", - }; - targetContext["tiles"] = "https://w3id.org/tree/terms#"; - targetContext["hydra"] = "http://www.w3.org/ns/hydra/core#"; - targetContext["hydra:variableRepresentation"] = { - "@type": "@id", - }; - targetContext["hydra:property"] = { - "@type": "@id", - }; - targetContext["sh"] = "https://www.w3.org/ns/shacl#"; - targetContext["sh:path"] = { - "@type": "@id", - }; - targetContext["ngsi-ld"] = "https://uri.etsi.org/ngsi-ld/"; -} - -function simplifyGraph(vocabulary, graph) { - // fixme; simple placeholder - for (const entity of graph) { - delete entity["@context"]; - } -} +} \ No newline at end of file diff --git a/src/controllers/latestController.ts b/src/controllers/latestController.ts index 5c0b49e..702ed74 100644 --- a/src/controllers/latestController.ts +++ b/src/controllers/latestController.ts @@ -6,12 +6,12 @@ import GeohashFragmenter from "../fragmenters/geohash"; import H3Fragmenter from "../fragmenters/h3"; import SlippyFragmenter from "../fragmenters/slippy"; import TimeFragmenter from "../fragmenters/time"; +import {convertResponseToEventstream, expandVocabulary, extractVocabulary, simplifyGraph} from "../utils/Util"; -// tslint:disable: no-string-literal - -function wrapLatest( +async function wrapLatest( req: Request, // the original request data: object, // the converted data + type: string, endDate: Date, // when was this known to be the latest data geoFragmenter: GeoFragmenter, // geospatial fragmentation strategy ) { @@ -20,9 +20,9 @@ function wrapLatest( const precision = geoFragmenter.getPrecision(req); // adapt/use a new json-ld context - const vocabulary = extractVocubalary(data); - expandVocabulary(vocabulary); - simplifyGraph(vocabulary, data); + let vocabulary = extractVocabulary(data); + vocabulary = await expandVocabulary(vocabulary); + // simplifyGraph(vocabulary, data); const config = getConfig(); @@ -32,7 +32,7 @@ function wrapLatest( const children = [{ "@type": "tree:AlternateViewRelation", - "tree:node": geoFragmenter.getDataFragmentURI(config.targetURI, focus, precision, beginTime), + "tree:node": geoFragmenter.getDataFragmentURI(config.targetURI, type, focus, precision, beginTime), "sh:path": "ngsi-ld:observedAt", "tree:value": { "schema:startDate": beginTime.toISOString(), @@ -43,17 +43,17 @@ function wrapLatest( // build the fragment const result = { "@context": vocabulary, - "@id": geoFragmenter.getLatestFragmentURI(config.targetURI, focus, precision), + "@id": geoFragmenter.getLatestFragmentURI(config.targetURI, type, focus, precision), "@type": "tree:Node", ...geoFragmenter.getMetaData(focus, precision), "tree:relation": children, - "sh:path": "ngsi-ld:observedAt", + "tree:path": "ngsi-ld:observedAt", "tree:value": { "schema:endDate": endDate.toISOString(), }, "dcterms:isPartOf": { "@id": config.targetURI, - "@type": "hydra:Collection", + "@type": "tree:Collection", "hydra:search": geoFragmenter.getLatestearchTemplate(config.targetURI), }, "@graph": data, @@ -79,6 +79,16 @@ async function getLatest( return; } + // type for NGSI-LD + if (!req.query.type) { + res.status(404).send("Provide \"type\" query parameter"); + return; + } + const type = decodeURIComponent(req.query.type.toString()); + + const config = getConfig(); + + const fromTime = new Date(new Date().getTime() - config.lastNumberOfMinutes * 60 * 1000); // going to fetch the most recent observations up until NOW const toTime = new Date(); @@ -87,15 +97,16 @@ async function getLatest( const focus = geoFragmenter.getFocusPoint(req); const bbox = [geoFragmenter.getBBox(focus, precision).map((location) => [location.longitude, location.latitude])]; - const config = getConfig(); - const uri = `${config.sourceURI}/temporal/entities?georel=within&geometry=Polygon&` - + `coordinates=${JSON.stringify(bbox)}&timerel=before` - + `&time=${toTime.toISOString()}&lastN=${config.lastN}`; + const uri = `${config.sourceURI}/temporal/entities?type=${encodeURIComponent(type)}&georel=within&geometry=Polygon&` + + `coordinates=${JSON.stringify(bbox)}&timerel=between` + + `&time=${fromTime.toISOString()}&endTime=${toTime.toISOString()}` + + `&timeproperty=modifiedAt&options=sysAttrs`; const response = await fetch(uri); - const data = await response.json(); + const responseJson = await response.json(); + const data = await convertResponseToEventstream(responseJson, type, fromTime, toTime, bbox); // add metadata to the resulting data - const wrappedData = wrapLatest(req, data, toTime, geoFragmenter); + const wrappedData = await wrapLatest(req, data, type, toTime, geoFragmenter); addHeaders(res); res.status(200).send(wrappedData); @@ -123,57 +134,3 @@ export async function getH3Latest(req, res) { await getLatest(req, res, geoFragmenter); } -/* - * Placeholders - * There are probably libraries that automate the mutating of contexts - * Ideally these functions create and use a new vocabulary combining the original data and the derived view - */ - -function extractVocubalary(data) { - // fixme; simple placeholder - if (data && data.length) { - return data[0]["@context"]; - } else { - return {}; - } -} - -function expandVocabulary(vocabulary) { - let targetContext; - if (vocabulary && vocabulary.length) { - targetContext = vocabulary[0]; - } else { - targetContext = vocabulary; - } - - targetContext["xsd"] = "http://www.w3.org/2001/XMLSchema#"; - targetContext["schema"] = "http://schema.org/"; - targetContext["schema:endDate"] = { - "@type": "xsd:dateTime", - }; - targetContext["dcterms"] = "http://purl.org/dc/terms/"; // to describe the dataset - targetContext["tiles"] = "https://w3id.org/tree/terms#"; // for the fragmentations - targetContext["hydra"] = "http://www.w3.org/ns/hydra/core#"; // for the hypermedia controls - targetContext["hydra:variableRepresentation"] = { - "@type": "@id", - }; - targetContext["hydra:property"] = { - "@type": "@id", - }; - targetContext["tree"] = "https://w3id.org/tree/terms#"; - targetContext["tree:node"] = { - "@type": "@id", - }; - targetContext["sh"] = "https://www.w3.org/ns/shacl#"; - targetContext["sh:path"] = { - "@type": "@id", - }; - targetContext["ngsi-ld"] = "https://uri.etsi.org/ngsi-ld/"; -} - -function simplifyGraph(vocabulary, graph) { - // fixme; simple placeholder - for (const entity of graph) { - delete entity["@context"]; - } -} diff --git a/src/fragmenters/GeoFragmenter.ts b/src/fragmenters/GeoFragmenter.ts index c71e8a5..b08cce9 100644 --- a/src/fragmenters/GeoFragmenter.ts +++ b/src/fragmenters/GeoFragmenter.ts @@ -32,20 +32,20 @@ export abstract class GeoFragmenter { public abstract getLatestearchTemplate(baseUri: string); /* Returns the paginated raw data URI */ - public getDataFragmentURI(base: string, focus: ILocation, precision: number, time?: Date) { + public getDataFragmentURI(base: string, type: string, focus: ILocation, precision: number, time?: Date) { const path = this.getFragmentPath(focus, precision); const geospatial = `${base}${path}`; if (time) { - return `${geospatial}?page=${time.toISOString()}`; + return `${geospatial}?page=${time.toISOString()}&type=${encodeURIComponent(type)}`; } else { - return geospatial; + return `${geospatial}?type=${encodeURIComponent(type)}`; } } /* Returns the latest data URI */ - public getLatestFragmentURI(base: string, focus: ILocation, precision: number) { + public getLatestFragmentURI(base: string, type: string, focus: ILocation, precision: number) { const path = this.getFragmentPath(focus, precision); const geospatial = `${base}${path}`; - return `${geospatial}/latest`; + return `${geospatial}/latest?type=${encodeURIComponent(type)}`; } /* Returns the paginated summary data URI */ public getSummaryFragmentURI(base: string, focus: ILocation, precision: number, time: Date, period: string) { diff --git a/src/fragmenters/slippy.ts b/src/fragmenters/slippy.ts index 30ecfc6..67c7eb4 100644 --- a/src/fragmenters/slippy.ts +++ b/src/fragmenters/slippy.ts @@ -60,7 +60,7 @@ export default class SlippyFragmenter extends GeoFragmenter { { longitude: lon1, latitude: lat1 }, { longitude: lon1, latitude: lat2 }, { longitude: lon2, latitude: lat2 }, - { longitude: lon2, latitude: lat2 }, + { longitude: lon2, latitude: lat1 }, ]; } @@ -85,7 +85,7 @@ export default class SlippyFragmenter extends GeoFragmenter { public getDataSearchTemplate(baseUri: string) { return { "@type": "hydra:IriTemplate", - "hydra:template": `${baseUri}/{z}/{x}/{y}{?page}`, + "hydra:template": `${baseUri}/{z}/{x}/{y}{?type,page}`, "hydra:variableRepresentation": "hydra:BasicRepresentation", "hydra:mapping": [ { @@ -106,12 +106,18 @@ export default class SlippyFragmenter extends GeoFragmenter { "hydra:property": "tiles:latitudeTile", "hydra:required": true, }, + { + "@type": "hydra:IriTemplateMapping", + "hydra:variable": "type", + "hydra:property": "rdf:type", + "hydra:required": true, + }, { "@type": "hydra:IriTemplateMapping", "hydra:variable": "page", "hydra:property": "schema:startDate", "hydra:required": false, - }, + } ], }; } diff --git a/src/fragmenters/time.ts b/src/fragmenters/time.ts index cb590c1..35e926c 100644 --- a/src/fragmenters/time.ts +++ b/src/fragmenters/time.ts @@ -1,3 +1,5 @@ +import * as QueryString from "qs"; + export default class TimeFragmenter { /* Some preset time ranges */ public static readonly HOUR = 1000 * 60 * 60; @@ -12,7 +14,7 @@ export default class TimeFragmenter { } /* Returns the start time of a fragment, given the time range the client requested */ - public getFromTime(requestedPage: string|Date): Date { + public getFromTime(requestedPage: string | string[] | QueryString.ParsedQs | QueryString.ParsedQs[] | Date): Date { let date: Date; let requestedTime: number; @@ -20,7 +22,7 @@ export default class TimeFragmenter { requestedTime = Date.now(); } else if (typeof requestedPage === "string") { requestedTime = Date.parse(decodeURIComponent(requestedPage)); - } else { + } else if (requestedPage instanceof Date) { requestedTime = requestedPage.getTime(); } diff --git a/src/routes/all.ts b/src/routes/all.ts index 441ba1b..c72eb26 100644 --- a/src/routes/all.ts +++ b/src/routes/all.ts @@ -5,10 +5,10 @@ import { getGeohashSummaryPage, getH3SummaryPage, getSlippySummaryPage } from ". const router = express.Router(); -router.get("/:zoom/:tile_x/:tile_y/summary", getSlippySummaryPage); -router.get("/geohash/:hash/summary", getGeohashSummaryPage); -router.get("/h3/:index/summary", getH3SummaryPage); - +// router.get("/:zoom/:tile_x/:tile_y/summary", getSlippySummaryPage); +// router.get("/geohash/:hash/summary", getGeohashSummaryPage); +// router.get("/h3/:index/summary", getH3SummaryPage); +// router.get("/:zoom/:tile_x/:tile_y/latest", getSlippyLatest); router.get("/geohash/:hash/latest", getGeohashLatest); router.get("/h3/:index/latest", getH3Latest); diff --git a/src/utils/Util.ts b/src/utils/Util.ts new file mode 100644 index 0000000..5c320f9 --- /dev/null +++ b/src/utils/Util.ts @@ -0,0 +1,168 @@ +import fetch = require("node-fetch"); +import { getConfig } from "../config/config"; +const ContextParser = require('jsonld-context-parser').ContextParser; + +function convertToKeyValues(entity: any) { + // Retrieve entities with options=keyValues + const converted = {}; + if (!(typeof entity === "object")) { return entity; } else { + for (const k in Object.keys(entity)) { + try { + if (entity[Object.keys(entity)[k]].type) { + if (entity[Object.keys(entity)[k]].type === "Relationship") { + converted[Object.keys(entity)[k]] = convertToKeyValues(entity[Object.keys(entity)[k]].object); + } else if (entity[Object.keys(entity)[k]].type === "Property" || entity[Object.keys(entity)[k]].type === "GeoProperty") { + converted[Object.keys(entity)[k]] = convertToKeyValues(entity[Object.keys(entity)[k]].value); + } else { + converted[Object.keys(entity)[k]] = convertToKeyValues(entity[Object.keys(entity)[k]]); + } + } else if (Array.isArray(entity[Object.keys(entity)[k]])) { + converted[Object.keys(entity)[k]] = []; + for (const a in entity[Object.keys(entity)[k]]) { + converted[Object.keys(entity)[k]].push(convertToKeyValues(entity[Object.keys(entity)[k]][a])); + } + } else if (typeof entity[Object.keys(entity)[k]] === 'object') { + converted[Object.keys(entity)[k]] = convertToKeyValues(entity[Object.keys(entity)[k]]); + } else { + converted[Object.keys(entity)[k]] = entity[Object.keys(entity)[k]]; + } + } catch (e) { + console.error("something went wrong with converting to keyValues. Continuing..."); + } + } + } + return converted; +} + +function getModifiedAtsFromEntity(modifiedAts: any[], object: any) { + for (const k in Object.keys(object)) { + // tslint:disable-next-line:max-line-length + if (Object.keys(object)[k] === "modifiedAt" && modifiedAts.indexOf(object[Object.keys(object)[k]]) === -1) { modifiedAts.push(object[Object.keys(object)[k]]); } + else if (Array.isArray(object[Object.keys(object)[k]])) { + for (const o in object[Object.keys(object)[k]]) { + getModifiedAtsFromEntity(modifiedAts, object[Object.keys(object)[k]][o]); + } + } else if (typeof object[Object.keys(object)[k]] === "object") { + getModifiedAtsFromEntity(modifiedAts, object[Object.keys(object)[k]]); + } + } +} + +function getModifiedAtsFromResponse(responseJson: any): any[] { + const modifiedAts = []; + if (Array.isArray(responseJson)) { + for (let e in responseJson) { + getModifiedAtsFromEntity(modifiedAts, responseJson[e]); + } + } else if (responseJson.id) { + // single entity + getModifiedAtsFromEntity(modifiedAts, responseJson); + } + return modifiedAts; +} + +function convertToEventStream(data: any[], type: string) { + const config = getConfig(); + + for (const d in data) { + data[d]["dcterms:isVersionOf"] = data[d].id; + // create version URI + data[d].id += "/" + new Date(data[d].modifiedAt).toISOString(); + data[d]["prov:generatedAtTime"] = new Date(data[d].modifiedAt).toISOString(); + data[d].memberOf = `${config.targetURI}/${encodeURIComponent(type)}`; + } + return data; +} + +export async function convertResponseToEventstream(responseJson: object, type: string, fromTime: Date, toTime: Date, bbox: object) { + const config = getConfig(); + + const modifiedAts = getModifiedAtsFromResponse(responseJson); + + const data = []; + + for (const ma in modifiedAts) { + const modifiedAt = new Date(modifiedAts[ma]); + if (modifiedAt.getTime() >= fromTime.getTime() && modifiedAt.getTime() < toTime.getTime()) { + const uriModifiedAt = `${config.sourceURI}/temporal/entities?type=${encodeURIComponent(type)}&georel=within&geometry=Polygon&` + + `coordinates=${JSON.stringify(bbox)}&timerel=between` + + `&time=${modifiedAt.toISOString()}&endTime=${modifiedAt.toISOString()}` + + `&timeproperty=modifiedAt&options=sysAttrs`; + const modifiedAtResponse = await fetch(uriModifiedAt); + const entities = await modifiedAtResponse.json(); + if (entities.id) { + if (Array.isArray(entities)) { + for (const e in entities) { + let entity = entities[e]; + // remove NGSI metadata model when keyValues is enabled + if (config.keyValues) entity = convertToKeyValues(entity); + data.push(entity); + } + } else { + let entity = entities; + if (config.keyValues) entity = convertToKeyValues(entities); + data.push(entity); + } + } + } + } + return convertToEventStream(data, type); +} + +export async function expandVocabulary(vocabulary) { + const myParser = new ContextParser(); + + let defaultContext = {}; + defaultContext["xsd"] = "http://www.w3.org/2001/XMLSchema#"; + defaultContext["schema"] = "http://schema.org/"; + defaultContext["schema:endDate"] = { + "@type": "xsd:dateTime", + }; + defaultContext["dcterms"] = "http://purl.org/dc/terms/"; // to describe the dataset + defaultContext["tiles"] = "https://w3id.org/tree/terms#"; // for the fragmentations + defaultContext["hydra"] = "http://www.w3.org/ns/hydra/core#"; // for the hypermedia controls + defaultContext["hydra:variableRepresentation"] = { + "@type": "@id", + }; + defaultContext["hydra:property"] = { + "@type": "@id", + }; + defaultContext["tree"] = "https://w3id.org/tree/terms#"; + defaultContext["tree:node"] = { + "@type": "@id", + }; + defaultContext["sh"] = "https://www.w3.org/ns/shacl#"; + defaultContext["tree:path"] = { + "@type": "@id", + }; + defaultContext["ngsi-ld"] = "https://uri.etsi.org/ngsi-ld/"; + + const myContext = await myParser.parse([defaultContext, vocabulary], { + external: true, + minimalProcessing: true // remote contexts are not ingested in our context + }); + + return myContext.getContextRaw(); +} + +/* + * Placeholders + * There are probably libraries that automate the mutating of contexts + * Ideally these functions create and use a new vocabulary combining the original data and the derived view + */ + +export function extractVocabulary(data) { + // fixme; simple placeholder + if (data && data.length) { + return data[0]["@context"]; + } else { + return {}; + } +} + +export function simplifyGraph(vocabulary, graph) { + // fixme; simple placeholder + for (const entity of graph) { + delete entity["@context"]; + } +} \ No newline at end of file