OpenSearch support for data discovery

[m-mohr]

Currently, we have a place holder for OpenSearch support in the API for data (and process) discovery. It bases on OpenSearch 1.0 and doesn't really fit into our API as it is XML based and not JSON as everything else.

It seems that OGC is working on a new JSON based draft as I could read here:

in the OGC EO-PMOS Standards Working Group we´re currently standardizing a new version of the OpenSearch EO-Extension (13-026r9 -> aligned with OWC Context, CEOS,...). Accompanied by a GeoJSON(-LD) response encoding (OGC 17-047), GeoJSON(-LD) based Metadata Profile for EO-products (granules) (OGC 17-003) and a GeoJSON(-LD) based Metadata Profile for EO-collections (just in draft status). We hope to have final version by 1/2018.

That sounds pretty interesting. They are overdue(?), but maybe we can just wait for the release and incorporate this version in favor of the XML-based version.

Can't really find further information, but there is a presentation from a CEOS meeting regarding the EO OpenSearch GeoJSON Encoding. There are some interesting information included, which we should look at, e.g. regarding on how to format the acquisition information that were proposed also in #64 for the general data discovery endpoint.

[m-mohr]

I removed OpenSearch from the API spec for now. It is not useful as of now and it's unclear whether we support it or at which endpoint it is implemented.

What has been removed:

  /data/opensearch:
    options:
      summary: Response to allow Cross-Origin Resource Sharing.
      description: >-
        Response for the preflight requests made by some clients due to
        Cross-Origin Resource Sharing restrictions. It sends the appropriate
        headers for this endpoint as defined in the section "Responses". See
        https://www.w3.org/TR/cors/ for more information.
      tags:
        - CORS
      responses:
        '204':
          $ref: '#/components/responses/cors_options'
        '405':
          $ref: '#/components/responses/method_not_allowed'
        '500':
          $ref: '#/components/responses/server_error'
    get:
      summary: OpenSearch endpoint to receive standardized data search results.
      description: >-
        This service offers more complex search functionality and returns
        results in an OpenSearch compliant Atom XML format.
      tags:
        - EO Data Discovery
      security:
        - {}
        - Bearer: []
      parameters:
        - in: query
          name: q
          description: string expression to search available datasets
          required: false
          schema:
            type: string
        - in: query
          name: start
          description: page start value
          required: false
          schema:
            type: integer
        - in: query
          name: rows
          description: page size value
          required: false
          schema:
            type: integer
      responses:
        '200':
          description: >-
            An array of EO datasets including their unique identifiers and some
            basic metadata.
          headers:
            Access-Control-Allow-Origin:
              $ref: '#/components/headers/allow_origin'
            Access-Control-Allow-Credentials:
              $ref: '#/components/headers/allow_credentials'
          content:
            application/atom+xml:
              examples:
                response:
                  value: >-
                    <?xml version="1.0" encoding="UTF-8"?> <feed
                    xmlns="http://www.w3.org/2005/Atom" 
                      xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">
                      <title>Supported EO datasets by Example, Inc.</title>
                      <updated>2018-01-18T10:30:00Z</updated>
                      <author>
                        <name>Example, Inc.</name>
                      </author> 
                      <id>urn:uuid:60a76c80-d399-11d9-b93C-0003939e0af6</id>
                      <link rel="self" href="http://www.exmaple.com/api/1.0/data/opensearch"/>
                      <link rel="search" type="application/opensearchdescription+xml" href="http://example.com/api/1.0/data/opensearch/description.xml"/>
                      <opensearch:totalResults>3</opensearch:totalResults>
                      <opensearch:Query role="request" startPage="1" />
                      <entry>
                        <title>MOD09Q1</title>
                        <link rel="alternate" href="http://www.exmaple.com/api/1.0/data/MOD09Q1"/>
                        <id>urn:uuid:1223c695-cfb8-4ebb-aaaa-80da344efa6a</id>
                        <updated>2017-09-18T10:30:00Z</updated>
                        <rights type="text">U.S. Geological Survey (USGS), DOI: 10.5067/MODIS/MOD09Q1.006</rights>
                        <summary type="text">MODIS/Terra Surface Reflectance 8-Day L3 Global 250m SIN Grid V00 </summary>
                      </entry>
                      <entry>
                        <title>SENTINEL2-1C</title>
                        <link rel="alternate" href="http://www.exmaple.com/api/1.0/data/SENTINEL2-1C"/>
                        <id>urn:uuid:1224c695-cfc8-4ebb-bbbb-90da344efa6b</id>
                        <updated>2017-12-01T01:00:00Z</updated>
                        <rights type="text">European Space Agency (ESA)</rights>
                        <summary type="text">Sentinel 2 Level-1C: Top-of-atmosphere reflectances in cartographic geometry</summary>
                      </entry>
                      <entry>
                        <title>LandsatETM+</title>
                        <link rel="alternate" href="http://www.exmaple.com/api/1.0/data/LandsatETM%2B"/>
                        <id>urn:uuid:1225c695-cfd8-4ebb-cccc-70da344efa6c</id>
                        <updated>2015-01-10T22:45:12Z</updated>
                        <rights type="text">U.S. Geological Survey (USGS)</rights>
                        <summary type="text">Landsat Enhanced Thematic Mapper Plus (ETM+)</summary>
                      </entry>
                    </feed>
        '400':
          $ref: '#/components/responses/client_error'
        '401':
          $ref: '#/components/responses/unauthorized'
        '403':
          $ref: '#/components/responses/forbidden'
        '500':
          $ref: '#/components/responses/server_error'
  /processes/opensearch:
    options:
      summary: Response to allow Cross-Origin Resource Sharing.
      description: >-
        Response for the preflight requests made by some clients due to
        Cross-Origin Resource Sharing restrictions. It sends the appropriate
        headers for this endpoint as defined in the section "Responses". See
        https://www.w3.org/TR/cors/ for more information.
      tags:
        - CORS
      responses:
        '204':
          $ref: '#/components/responses/cors_options'
        '405':
          $ref: '#/components/responses/method_not_allowed'
        '500':
          $ref: '#/components/responses/server_error'
    get:
      summary: OpenSearch endpoint to request standardized process search results.
      description: >-
        This service offers more complex search functionality and returns
        results in an OpenSearch compliant Atom XML format.
      tags:
        - Process Discovery
      security:
        - {}
        - Bearer: []
      parameters:
        - name: q
          in: query
          description: string expression to search available processes
          required: false
          schema:
            type: string
        - name: start
          in: query
          description: page start value
          required: false
          schema:
            type: integer
        - name: rows
          in: query
          description: page size value
          required: false
          schema:
            type: integer
      responses:
        '200':
          description: OpenSearch response
          headers:
            Access-Control-Allow-Origin:
              $ref: '#/components/headers/allow_origin'
            Access-Control-Allow-Credentials:
              $ref: '#/components/headers/allow_credentials'
          content:
            application/atom+xml:
              examples:
                response:
                  value: >-
                    <?xml version="1.0" encoding="UTF-8"?> <feed
                    xmlns="http://www.w3.org/2005/Atom" 
                      xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">
                      <title>Supported EO processes by Example, Inc.</title>
                      <updated>2018-01-18T10:30:00Z</updated>
                      <author>
                        <name>Example, Inc.</name>
                      </author> 
                      <id>urn:uuid:60a76c80-d399-11d9-b93C-0003939e0af6</id>
                      <link rel="self" href="http://www.exmaple.com/api/1.0/processes/opensearch"/>
                      <link rel="search" type="application/opensearchdescription+xml" href="http://example.com/api/1.0/processes/opensearch/description.xml"/>
                      <opensearch:totalResults>2</opensearch:totalResults>
                      <opensearch:Query role="request" startPage="1" />
                      <entry>
                        <title>NDVI</title>
                        <link rel="alternate" href="http://www.exmaple.com/api/1.0/processes/NDVI"/>
                        <id>urn:uuid:1223c695-cfb8-4ebb-aaaa-80da344efa6a</id>
                        <updated>2018-01-18T10:30:00Z</updated>
                        <summary type="text">Computes the normalized difference vegetation index (NDVI) for all pixels of the input dataset.</summary>
                      </entry>
                      <entry>
                        <title>median_time</title>
                        <link rel="alternate" href="http://www.exmaple.com/api/1.0/data/median_time"/>
                        <id>urn:uuid:1224c695-cfc8-4ebb-bbbb-90da344efa6b</id>
                        <updated>2018-01-18T10:30:00Z</updated>
                        <summary type="text">Applies median aggregation to pixel time series for all bands of the input dataset.</summary>
                      </entry>
                    </feed>
        '400':
          $ref: '#/components/responses/client_error'
        '401':
          $ref: '#/components/responses/unauthorized'
        '403':
          $ref: '#/components/responses/forbidden'
        '500':
          $ref: '#/components/responses/server_error'
```

[m-mohr]

I removed process discovery as I don't think OpenSearch makes sense in this context and make things more complex than they need to be.

[m-mohr]

I just removed (see 9091559) the query parameters for the data discovery endpoint as they most likely will be changed later in favor of either a standard-specific version of the body contents (e.g. STAC) or OpenSearch.

[m-mohr]

I just had a brief look into the most recent version of OGC OpenSearch EO, incl. the new JSON encoding. I found it very hard to really get into it due to its massive length. Some comments follow...

Most interesting is probably the search functionality:

• Minimum set of search parameters they want back-ends to support: geo:uid, geo:box, time:start (not datetime, see Avoid name in favor of id and title #130), time:end
• Optionally, servers can support very advanced syntax, e.g. CQL.
• Collection search can search for productType, doi, platform, sensor, compositeType (a time period), processing level, orbit type, spectral range, wavelength, ...
• Additional INSPIRE obligated parameters for collection search: title, category, keyword, abstract, organizationName, publicationDate, lineage, contraints, language, ...
• Product search has a ton (> 40) of parameters, I won't name them all, but they even include some really specific stuff like dopplerFrequency.

Regarding the JSON response format:

• Uses GeoJSON as basis, no other options.

Preliminary conclusion: Too complex for us. Would skip it for now and just take the easier things we inherit from using STAC.

[m-mohr]

Now that we incorporate STAC and WFS into the data discovery, it's time to make a short wrap-up regarding OpenSearch in openEO.

I already listed some information in my previous comments and there were some related discussions also in a STAC issue (radiantearth/stac-spec#16). In the end, OpenSearch is very complex, it's like a 300 pages that are exceeding the current documentation of the whole openEO project, a better start is probably the CEOS OpenSearch best practice document. OpenSearch enforces a different JSON encoding for the responses, which is not compatible with STAC. So that's something that could not really be incorporated into our existing endpoints. I see it more to be put on top as a separate endpoint/extension, but I don't currently see in the core. It may be added to STAC or WFS as extension, but not even WFS as a OGC standard has included their own OGC OpenSearch extension, so I don't really see a point why we should push it into the core. Maybe we could work that out as an openEO API extension by interested parties/partners.

[m-mohr]

As discussed in #103, we won't go for OpenSearch support at the moment. We may reconsider it in the future.