Merge pull request #170 from Open-EO/0.4.0

openEO API v0.4.0

Author: m-mohr

.gitmodules

@@ -0,0 +1,3 @@
+[submodule "processes"]
+	path = processes
+	url = https://github.com/Open-EO/openeo-processes

.travis.yml

@@ -13,18 +13,21 @@ before_install:
 - pip install mkdocs-macros-plugin
 - pip install pymdown-extensions
 - pip install pygments
-- npm install -g asyncapi-docgen
+- npm install -g asyncapi-generator
+- npm install -g concat-json-files
+- npm install -g speccy
+- speccy lint openapi.json -r "speccy.yml"
 
 install:
+- concat-json-files "processes/*.json" -t "processes.json"
 - mv CHANGELOG.md docs/changelog.md
+- ag subscriptions.json markdown
+- \mv -f asyncapi.md docs/apireference-subscriptions.md
 - mkdocs build --clean
 - cp *.json site/
-- rm -f site/apireference/index.html
-- rm -f site/processreference/index.html
-- rm -f site/apireference-subscriptions/index.html
-- mv docs/apireference.html site/apireference/index.html
-- mv docs/processreference.html site/processreference/index.html
-- adg subscriptions.json -o site/apireference-subscriptions
+- \mv -f docs/apireference.html site/apireference/index.html
+- \mv -f docs/processreference.html site/processreference/index.html
+- cp processes/*.json site/processreference/
 
 script:
 - git clone --branch gh-pages https://$GITHUB_TOKEN@github.com/Open-EO/openeo-api.git gh-pages

CHANGELOG.md

@@ -4,14 +4,75 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.4.0] - 2019-03-07
+### Added
+- `GET /jobs/{job_id}/estimate` can return the estimated required storage capacity. [#122](https://github.com/Open-EO/openeo-api/issues/122)
+- `GET /jobs/{job_id}` has two new properties:
+    - `progress` indicates the batch job progress when running. [#82](https://github.com/Open-EO/openeo-api/issues/82)
+    - `error` states the error message when a job errored out.
+      `GET /jobs/{job_id}/result` mirrors this error message in a response with HTTP status code 424. [#165](https://github.com/Open-EO/openeo-api/issues/165)
+- `GET /.well-known/openeo` allows clients to choose between versions. [#148](https://github.com/Open-EO/openeo-api/issues/148)
+- `GET /` (Capabilities):
+    - Requires to return a title (`title`), a description (`description`) and the back-end version (`backend_version`). [#154](https://github.com/Open-EO/openeo-api/issues/154)
+    - Billing plans have an additional required property `paid`. [#157](https://github.com/Open-EO/openeo-api/issues/157)
+    - Should provide a link to the Well-Known URI (`/.well-known/openeo`) in the new `links` property.
+- `GET /processes` (Process discovery):
+    - Processes can be categorizes with the `category` property.
+    - Parameters can be ordered with the `parameter_order` property instead of having a random order.
+    - Support for references to other processes in descriptions.
+    - Processes and parameters can be declared to be `experimental`.
+- `GET /output_formats` and `GET /service_types` can now provide links per entry.
+- `GET /udf_runtimes` provide a list of UDF runtime environments. [#87](https://github.com/Open-EO/openeo-api/issues/87)
+- `GET /service_types` allows to specify `variables` that can be used in process graphs. [#172](https://github.com/Open-EO/openeo-api/issues/172)
+
+### Changed
+- Completely new version of the processes.
+- Changed process graph to a flexible graph-like structure, which also allows callbacks. [#160](https://github.com/Open-EO/openeo-api/issues/160)
+- Updated `GET /collections` and `GET /collections/{collection_id}` to follow STAC v0.6.2. [#158](https://github.com/Open-EO/openeo-api/issues/158), [#173](https://github.com/Open-EO/openeo-api/issues/173)
+- The `process_graph_id` of stored process graphs, the `service_id` of services and the `job_id` of jobs has changed to `id` in responses. [#130](https://github.com/Open-EO/openeo-api/issues/130)
+- The `status` property for jobs is now required. 
+- `POST /preview` renamed to `POST /result`. [#162](https://github.com/Open-EO/openeo-api/issues/162)
+- `GET /` (Capabilities):
+    - `version` in the response was renamed to `api_version`.
+    - Endpoint paths must follow the openAPI specification. [#128](https://github.com/Open-EO/openeo-api/issues/128)
+    - Billing plan descriptions allow CommonMark. [#164](https://github.com/Open-EO/openeo-api/issues/164)
+- `/files/{user_id}/{path}` File management:
+    - Clarified handling of folders. [#146](https://github.com/Open-EO/openeo-api/issues/146)
+    - `GET` method: The `name` property was renamed to `path`. [#133](https://github.com/Open-EO/openeo-api/issues/133)
+    - `PUT` method: Returns file meta data with a different response code. [#163](https://github.com/Open-EO/openeo-api/issues/163)
+- `GET /processes` (Process discovery):
+    - The `name` property of processes has changed to `id`. [#130](https://github.com/Open-EO/openeo-api/issues/130)
+    - `mime_type` replaced with `media_type` in the input parameters and return values.
+    - The schema for `exceptions` follows the general schema for openEO errors. [#139](https://github.com/Open-EO/openeo-api/issues/139)
+    - Changed the structure of `examples`.
+- `POST /validation` (Process graph validation):
+    - Returns HTTP status code 200 for valid and invalid process graphs and responds with a list of errors. [#144]
+    - Allowed to call the endpoint without authentication. [#151](https://github.com/Open-EO/openeo-api/issues/151)
+- Behavior for `DELETE /jobs/{job_id}/results` and `POST /jobs/{job_id}/results` specified depending on the job status. Clarified status changes in general. [#142](https://github.com/Open-EO/openeo-api/issues/142)
+- Improved client development guidelines. [#124](https://github.com/Open-EO/openeo-api/issues/124), [#138](https://github.com/Open-EO/openeo-api/issues/138)
+
+### Removed
+- Numeric openEO error codes. Replaced in responses with textual error codes. [#139](https://github.com/Open-EO/openeo-api/issues/139)
+- Query parameters to replace process graph variables in `GET /process_graphs/{process_graph_id}`. [#147](https://github.com/Open-EO/openeo-api/issues/147)
+- `min_parameters` and `dependencies` for parameters in process descriptions returned by `GET /processes`.
+- Replaced output format properties in favor of an export process, which has resulted in in the removal of:
+  - The default output format in `GET /output_formats`. [#153](https://github.com/Open-EO/openeo-api/issues/153)
+  - The output format properties in `POST /result` (fka `POST /preview`), `POST /jobs`, `PATCH /jobs` and `GET /jobs/{job_id}` requests. [#153](https://github.com/Open-EO/openeo-api/issues/153)
+  - `gis_data_type` (not to be confused with `gis_data_types`) in the parameters of output formats in `GET /output_formats`
+
+### Fixed
+- Added missing `Access-Control-Expose-Headers` header to required CORS headers.
+- Some endpoints didn't include authentication information.
+- `GET /jobs/{job_id}/estimate`: Property `downloads_included` had a wrong default value.
+
 ## [0.3.1] - 2018-11-06
 
 ### Added
 - `createProcessGraph` method to client development guidelines.
 - JSON file with all specified errors.
 - Textual error codes for each specified error.
 - Allow setting a plan for `POST /preview`
-- Default billing plan in `GET /`.
+- Default billing plan in `GET /`. [#141](https://github.com/Open-EO/openeo-api/issues/141)
 - Job ID in JSON response for `GET /jobs/{job_id}/results`.
 
 ### Changed

README.md

@@ -2,29 +2,31 @@
 
 openEO develops an open API to connect R, python and javascript clients to big Earth observation cloud back-ends in a simple and unified way. This repository contains this API, the openEO (core) API.
 
-* **[Documentation / Specification](https://open-eo.github.io/openeo-api/v/0.3.1/index.html)**
+* **[Documentation / Specification](https://open-eo.github.io/openeo-api/v/0.4.0/index.html)**
 
 ## Versions
 
-The openEO (core) API is currently released in version **0.3.1**.
+The openEO (core) API is currently released in version **0.4.0**.
 
 **Note:** The specification is currently still an early version, with the potential for some major things to change. The core is now fleshed out, so implementors are encouraged to try it out and give feedback. But the goal is to actually be able to act on that feedback, which will mean changes are quite possible. A solid basis is specified right now, but best practices, extensions and specification details will emerge with implementation.
 
-| Version / Branch                                             | Status  | Description                                                  |
-| ------------------------------------------------------------ | ------- | ------------------------------------------------------------ |
-| [0.0.1](https://github.com/Open-EO/openeo-api/tree/0.0.1) ([Spec](https://open-eo.github.io/openeo-api/v/0.0.1/index.html)) | legacy  | First draft with basic ideas, loosely implemented.           |
-| [0.0.2](https://github.com/Open-EO/openeo-api/tree/0.0.2) ([Spec](https://open-eo.github.io/openeo-api/v/0.0.2/index.html)) | legacy  | Proof of concept, implemented.                               |
+| Version / Branch                                             | Status  | Description |
+| ------------------------------------------------------------ | ------- | ----------- |
+| [0.0.1](https://github.com/Open-EO/openeo-api/tree/0.0.1) ([Spec](https://open-eo.github.io/openeo-api/v/0.0.1/index.html)) | legacy  | First draft with basic ideas, loosely implemented. |
+| [0.0.2](https://github.com/Open-EO/openeo-api/tree/0.0.2) ([Spec](https://open-eo.github.io/openeo-api/v/0.0.2/index.html)) | legacy  | Proof of concept, implemented. |
 | [0.3.0](https://github.com/Open-EO/openeo-api/tree/0.3.0) ([Spec](https://open-eo.github.io/openeo-api/v/0.3.0/index.html)) | legacy | Major rework. |
-| [**0.3.1**](https://github.com/Open-EO/openeo-api/tree/0.3.1) ([Spec](https://open-eo.github.io/openeo-api/v/0.3.1/index.html)) | **current** | Fixing minor issues, see the changelog. |
-| [0.4.0](https://github.com/Open-EO/openeo-api/tree/0.4.0) ([Spec](https://open-eo.github.io/openeo-api/v/0.4.0/index.html)) | planned | Planned to improve data set descriptions, add process specifications and an extension concept. |
+| [0.3.1](https://github.com/Open-EO/openeo-api/tree/0.3.1) ([Spec](https://open-eo.github.io/openeo-api/v/0.3.1/index.html)) | legacy, supported | Fixing minor issues, see the [changelog](CHANGELOG.md#031---2018-11-06). |
+| [**0.4.0**](https://github.com/Open-EO/openeo-api/tree/0.4.0) ([Spec](https://open-eo.github.io/openeo-api/v/0.4.0/index.html)) | **current** | Improved discovery, added processes catalogue, new process graph structure and [more](CHANGELOG.md#031---2019-02-28). |
+| [0.5.0](https://github.com/Open-EO/openeo-api/tree/0.5.0) ([Spec](https://open-eo.github.io/openeo-api/v/0.5.0/index.html)) | planned | Improvements based on implementer feedback, introduce extension concept. |
 
 See also the [changelog](CHANGELOG.md) and the [milestones](https://github.com/Open-EO/openeo-api/milestones) for a rough roadmap based on GitHub issues.
 
 ## Repository
 
 This repository contains a set of files formally and technically describing the openEO API, each with a human-readable and easily browseable version:
 
-* [openapi.json](openapi.json) ([browseable version](https://open-eo.github.io/openeo-api/v/0.3.1/apireference/)) provides the [openAPI](https://www.openapis.org/) 3.0 definition of the openEO API.
-* [processes.json](processes.json) ([browseable version](https://open-eo.github.io/openeo-api/v/0.3.1/processreference/)) defines pre-defined core processes back-ends may implement for best interoperability.
-* [subscriptions.json](subscriptions.json) ([browseable version](https://open-eo.github.io/openeo-api/v/0.3.1/apireference-subscriptions/)) provides the [AsyncAPI](https://www.asyncapi.com/) 1.2 definitions for the WebSocket-based subscriptions and notifications API for openEO.
-* [docs/](docs/) ([browseable version](https://open-eo.github.io/openeo-api/v/0.3.1/)) contains all additional written documentation, including 'getting started' guides, the architecture, feature descriptions, development guidelines and more.
+* [docs/](docs/) ([browseable version](https://open-eo.github.io/openeo-api/v/0.4.0/)) contains all additional written documentation, including 'getting started' guides, the architecture, feature descriptions, development guidelines and more.
+* [processes/](processes/) ([browseable version](https://open-eo.github.io/openeo-api/v/0.4.0/processreference/)) defines pre-defined core processes back-ends may implement for best interoperability.
+* [openapi.json](openapi.json) ([browseable version](https://open-eo.github.io/openeo-api/v/0.4.0/apireference/)) provides the [openAPI](https://www.openapis.org/) 3.0 definition of the openEO API.
+* [subscriptions.json](subscriptions.json) ([browseable version](https://open-eo.github.io/openeo-api/v/0.4.0/apireference-subscriptions/)) provides the [AsyncAPI](https://www.asyncapi.com/) 1.2 definitions for the WebSocket-based subscriptions and notifications API for openEO.
+* [errors.json](errors.json) ([browseable version](https://open-eo.github.io/openeo-api/v/0.4.0/errors/#openeo-error-codes)) is a list of potential global error codes and messages, excluding specific exceptions separately available for each process.

docs/apireference.html

@@ -1,17 +1,12 @@
-<html lang="en" class="js json svg checked target dataset details fetch supports csstransforms3d no-ios" style="">
+<html lang="en">
   <head>
     <meta charset="utf-8">
     <meta name="viewport" content="width=device-width,initial-scale=1">
     <meta http-equiv="x-ua-compatible" content="ie=edge">
     <link rel="shortcut icon" href="../assets/images/favicon.png">
     <title>openEO API Reference</title>
-    <link rel="stylesheet" href="../assets/stylesheets/application.451f80e5.css">
+    <link rel="stylesheet" href="../img/header.css">
     <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:300,400,400i,700|Roboto+Mono">
-    <style>
-      body,input{font-family:"Roboto","Helvetica Neue",Helvetica,Arial,sans-serif}
-      code,kbd,pre{font-family:"Roboto Mono","Courier New",Courier,monospace}
-      .md-header-nav__button.md-logo {display: inline-block}
-    </style>
     <link rel="stylesheet" href="../assets/fonts/material-icons.css">
   </head>
   <body dir="ltr" data-md-state="">

docs/arch.md

@@ -11,20 +11,20 @@ The openEO API defines a language how clients communicate to back-ends in order
 7. The openEO API may define *profiles* in order to group specific functionality.
 
 ![Architecture - openEO API shown in dark blue](img/arch.png)
-*Figure: Architecture - openEO API shown in dark blue*
+*Figure: Architecture*
 
 # Microservices
 
 To simplify and structure the development, the API is divided into a few microservices.
 
-| Microservice                  | Description                                                  |
-| ----------------------------- | ------------------------------------------------------------ |
+| Microservice                  | Description |
+| ----------------------------- | ----------- |
 | Capabilities                  | This microservice reports on the capabilities of the back-end, i.e. which API endpoints are implemented, which authentication methods are supported, and whether and how UDFs can be executed at the back-end. |
-| EO Data Discovery             | Describes which datasets and image collections are available at the back-end. |
+| EO Data Discovery             | Describes which collections are available at the back-end. |
 | Process Discovery             | Provides services to find out which processes a back-end provides, i.e., what users can do with the available data. |
 | UDF                           | Discovery and execution of user-defined functions. |
-| Job Management                | Organizes and manages jobs that run processes on back-ends. |
+| Batch Job Management          | Organizes and manages batch jobs that run processes on back-ends. |
 | File Management               | Organizes and manages user-uploaded files. |
-| Process Graph Management      | Organizes and manages user-defined process graphs.  |
+| Process Graph Management      | Organizes and manages user-defined process graphs. |
 | Secondary Services Management | External web services to access data and job results such as a OGC WMTS service. |
 | Account Management            | User management, accounting and authentication. |