Data discovery

[pietercolpaert]

Challenge Description

When

• starting a new project, or
• moving an existing project into another domain, or
• maintaining your project for a long period of time,

you somehow need to find interoperable, relevant and trustworthy datasets. Today, this is a manual task. Automating this task requires a discovery mechanism, which on the Web today is an unsolved problem.

Example cases:

1. Setting up a new route planner.
2. Moving digital twin software from one city to another.
3. Creating a dashboard of a certain indicator, adding more data when it becomes available.

Impact and Importance

Automating data discovery should reduce the costs for:

1. setting up a new project
2. bringing the project into another context
3. maintaining the project over time

Desired Solution

1. A language to express the criteria for a dataset to enter your project, based on: the shape or schema used (e.g., SHACL), the provenance (e.g., only datasets that originate from X or Y), geo-temporal extent, usage conditions, etc.
2. A data model for Web-based storage system or data catalog so that the criteria can be evaluated.
3. An algorithm to evaluate 1 over 2

Acceptance Criteria

1. A specification is available of the language with examples on how to express datasets relevant to your application
2. A data model specification is available
3. A reference implementation of the algorithm can be tested

References and Resources

• Describing a data catalogs and their datasets with e.g., geotemporal aspects: DCAT
• Describing policies: ODRL
• For pipelines and provenance, see Pipelining workflows across participants #1

[TallTed]

SeeAlso: Describing Linked Datasets with the VoID Vocabulary

[rohitadeshmukh13]

Some more relevant references:

• SEMIC blog: Application Profiles: What are they and how to model and reuse them properly? A look through the DCAT-AP example.
• DCAT-AP 3.0
• DSSC Blueprint: Technical Building Blocks - especially the "Publication and Discovery (v1.5)" building block.

[JohannesLipp]

AFAIK, (distributed) queries on the data itself are not possible in dataspaces, because of the "keep data at source" paradigm and the data access/usage policies.

Therefore, one would need to explicitly represent the information needed for such a matching on the dataspace level, using metadata. This mainly includes metadata fields that give insights about the actual data content. In addition to VoID, which @TallTed already posted, the RDF Data Cube Vocabulary could represent insights about the data structure of data.

[TallTed]

AFAIK, (distributed) queries on the data itself are not possible in dataspaces, because of the "keep data at source" paradigm and the data access/usage policies.

Rather, the results of such (distributed) queries will vary with the querier, the dataspaces of which they are aware, the permissions they have thereon, etc.

Federated SPARQL (formerly known as SPARQL-FED) is one tool relevant to this practice. Writing a query that acts on multiple dataspaces/datastores requires knowledge of those spaces/stores, which can be gained along the way -- e.g., you start knowing about DataSpaceA, and run a query there, which results include references to DataSpaceB and DataSpaceC; then you run some "discovery" queries against DataSpaceB and DataSpaceC, and use the results of these queries to write a new query that targets all three of DataSpaceA, DataSpaceB, and DataSpaceC; these results include references to DataSpaceD and DataSpaceE, etc.

[VladimirAlexiev]

• @TallTed and @pietercolpaert AFAIK there are no EU dataspaces that hold semantic data so SHACL, VOID and SPARQL Federation doesn't apply. (The ERA KG is not a dataspace, not yet anyway)
• The only potential exception is the EU Grid digital twin and Energy data space (the electrical part), since electrical CIM/CGMES are 20-year old semantic standards. I only started hearing this year that the Energy data space focuses on CIM
• @pietercolpaert and @rohitadeshmukh13 DCAT is only the tip of the iceberg. Data spaces take the stance that ontologies and schemas are kept in a Vocabulary Hub and DCAT can refer to them using dct:conformsTo. But imho that's not enough: linking a dataset to some ontologies doesn't tell you much about its internal structure
• @rohitadeshmukh13 don't you find Publication and Discovery (v1.5) rather watered-down?
• @JohannesLipp Could you share an example how QB can be used to describe the structure of a tabular dataset?

In UNDERPIN, we use CSVW (csvw:Schema and csvw:Column) to describe tabular datasets and column characteristics (quantity, unit, features, qualifiers; datatype, format), which enables

• Rich semantic search using nested facets, eg "give me datasets that have readings of bearing temperature standardDeviations" translates to "datasets with columns where the 3 characteristics "feature, quantity, qualifier" co-occur
• Influx ingest, including individual CSVs into a DatasetSeries
• Flexible predictive analytics: rather than using hard-coded CSVs and column names, discover columns of interest

Here's the semantic model

• datasets
  
• schemas and columns
  
• relations
  

[VladimirAlexiev]

And here's the semantic search

[TallTed]

[@VladimirAlexiev] @TallTed and @pietercolpaert AFAIK there are no EU dataspaces that hold semantic data so SHACL, VOID and SPARQL Federation doesn't apply. (The ERA KG is not a dataspace, not yet anyway)

It's amazing what Google can reveal...

Cellar is the common data repository of the Publications Office of the European Union.

Cellar has an open SPARQL endpoint.

There are many other data spaces in the EU.