Skip to content

Commit e4f607a

Browse files
m-mohrsoxofaan
m-mohr
and
soxofaan
authored
Processing Parameters Extension #276 #545 (#471)
* Jobs and services: Added `create_parameters` property in responses to define the parameters that can be submitted suring creation #276 * Add processing options to a separate endpoint #276 * Options should be optional * Move to an extension * Add UDP extension * Apply suggestions from code review Co-authored-by: Stefaan Lippens <soxofaan@users.noreply.github.com> * Renamed according to PR review * Clarify where UDPs can come from * Apply suggestions from code review * Add example * Update extensions/processing-parameters/README.md Co-authored-by: Stefaan Lippens <soxofaan@users.noreply.github.com> --------- Co-authored-by: Stefaan Lippens <soxofaan@users.noreply.github.com>
1 parent 642f4d6 commit e4f607a

File tree

7 files changed

+217
-3
lines changed

7 files changed

+217
-3
lines changed

‎CHANGELOG.md

+1
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
1010

1111
- **New extensions:**
1212
- [Remote Process Definition Extension](./extensions/remote-process-definition/README.md)
13+
- [Processing Parameters Extension](./extensions/processing-parameters/README.md)
1314
- Added `version` property to `GET /processes` [#517](https://github.com/Open-EO/openeo-api/issues/517)
1415
- `POST /result`: Added response header "OpenEO-Identifier" to expose an identifier associated with a synchronous processing request.
1516

‎README.md

+1
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ See also the [changelog](CHANGELOG.md) and the [milestones](https://github.com/O
3333
| ------------------------------------------------------------------ | ------- | ------------ | ----------- |
3434
| [Commercial Data](extensions/commercial-data/) | 0.1.0 | experimental | Provides an interface for discovering, ordering and using commercial data. |
3535
| [Federation](extensions/federation/) | 0.1.0 | experimental | Covers federation aspects, i.e. where multiple back-ends are exposed as a single API. |
36+
| [Processing Parameters](extensions/processing-parameters/) | 0.1.0 | experimental | Explore and handle additional processing parameters that a back-end can offer for the processing modes (sync. processing, batch jobs, web services). |
3637
| [Remote Process Definition](extensions/remote-process-definition/) | 0.1.0 | experimental | Load user-defined processes that are hosted externally through the process namespace into process graphs. |
3738

3839
## Repository

‎extensions/processing-parameters/.spectral.yml

+12
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
extends: "spectral:oas"
2+
rules:
3+
contact-properties: true
4+
tag-description: true
5+
oas3-parameter-description: true
6+
oas3-unused-component: true
7+
operation-id-kebab-case:
8+
given: "$..operationId"
9+
then:
10+
function: pattern
11+
functionOptions:
12+
match: ^[a-z][a-z0-9\-]*$

‎extensions/processing-parameters/README.md

+75
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
# Processing Parameters Extension
2+
3+
The Processing Parameters Extension to the openEO API provides an interface to explore and handle additional processing parameters that a back-end can offer for the three processing modes (synchronous processing, batch jobs, secondary web services).
4+
5+
- Version: **0.1.0**
6+
- Stability: **experimental**
7+
- [OpenAPI document](openapi.yaml)
8+
- Conformance class: `https://api.openeo.org/extensions/processing-parameters/0.1.0`
9+
10+
**Note:** This document only documents the additions to the specification.
11+
Extensions can not change or break existing behavior of the openEO API.
12+
13+
This extension adds a new endpoint (`GET /processing_parameters`, see [OpenAPI document](openapi.yaml))
14+
to discover the additional processing parameters that a back-end offers.
15+
16+
Additionally, this extension allows to provide specific default values for user-defined processes (UDPs),
17+
which includes:
18+
19+
- UDPs submitted directly for synchronous processing, as batch jobs, or as secondary web services
20+
- UDPs stored through the `/process_graphs` endpoints
21+
- UDPs stored external to the API and retrieved through the [Remote Process Definition Extension](../remote-process-definition/README.md)
22+
23+
The parameters and its values are provided separately for each processing mode.
24+
25+
## Embedding default processing options in UDPs
26+
27+
UDPs can provide default values for specific processing parameters.
28+
29+
The values for each parameter (so called 'options') are provided separately for each processing mode.
30+
The following properties are added to the top-level of a UDP (e.g. as sibling nodes to the "process_graph" property) for the respective processing modes:
31+
32+
- `default_synchronous_options` for synchronous processing
33+
- `default_job_options` for batch jobs
34+
- `default_service_options` for secondary web services
35+
36+
The schema for each of these properties is:
37+
38+
```yaml
39+
type: object
40+
additionalProperties:
41+
description: Any type
42+
```
43+
44+
The keys of the object are the respective parameter names.
45+
The values of the object are the default values for the parameters.
46+
Schematic restrictions are not defined for the object, but the schemas for the parameters as defined in `GET /processing_parameters` apply to the given values.
47+
These values provide the defaults unless a user overrides them in the actual data processing request (e.g. `POST /jobs`, see below).
48+
49+
Example UDP including defaults for the processing parameters `memory` and `block-sizes` of a batch job:
50+
51+
```json
52+
{
53+
"id": "my_evi",
54+
"parameters": [...],
55+
"process_graph": {...},
56+
"default_job_options": {
57+
"memory": "2GB",
58+
"block-sizes": [128, 32]
59+
}
60+
}
61+
```
62+
63+
## Resolving parameters
64+
65+
Due to the variety of places where processing parameters can be provided, the following
66+
list defines how the parameters must be resolved. The prioritization is as follows:
67+
68+
1. If present, use the parameter specified in the processing request directly (e.g. in `POST /jobs` as a top-level property)
69+
2. If present, use the default parameter specified in the UDP
70+
3. Otherwise, use the default value for the parameter as specified in `GET /processing_parameters`
71+
72+
"Present" means that the property is present in the JSON representation regardless of the value given, i.e.
73+
properties are present if an empty string, an empty array, an empty object, `false`, `0`, or `null` are provided.
74+
75+
Unrecognized/unknown parameters that are provided through UDPs must be ignored by backends.

‎extensions/processing-parameters/openapi.yaml

+100
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
openapi: 3.0.2
2+
info:
3+
title: openEO API - Processing Parameters Extension
4+
version: 0.1.0
5+
description: >-
6+
The Processing Parameters Extension to the openEO API provides an interface to explore and handle additional processing parameters that a back-end can offer for the three processing modes (synchronous processing, batch jobs, and secondary web services).
7+
The openEO API specification allows back-ends to accept "additional back-end specific properties" for each of the processing modes.
8+
As the openEO API specification does not define the name and schemas of these properties, this extension provides a standardized way to define and document these properties.
9+
contact:
10+
name: openEO Consortium
11+
url: 'https://openeo.org'
12+
email: openeo.psc@uni-muenster.de
13+
license:
14+
name: Apache 2.0
15+
url: 'http://www.apache.org/licenses/LICENSE-2.0.html'
16+
externalDocs:
17+
url: https://github.com/Open-EO/openeo-api/blob/draft/extensions/workspaces/README.md
18+
tags:
19+
- name: Capabilities
20+
description: General information about the API implementation and other supported capabilities provided by the back-end.
21+
- name: Data Processing
22+
description: Organizes and manages data processing on the back-end, either as synchronous on-demand computation or batch jobs.
23+
- name: Batch Jobs
24+
description: Management of batch processing tasks (jobs) and their results.
25+
- name: Secondary Services
26+
description: On-demand access to data using other web service protocols.
27+
servers:
28+
- url: 'https://openeo.example/api/{version}'
29+
description: >-
30+
The URL of the API MAY freely be chosen by the back-end providers. The
31+
path, including API versioning, is a *recommendation* only. Nevertheless,
32+
all servers MUST support HTTPS as the authentication methods are not
33+
secure with HTTP only!
34+
variables:
35+
version:
36+
default: v1
37+
description: >-
38+
API versioning is RECOMMENDED. As the openEO API is following
39+
[SemVer](https://semver.org/) only the MAJOR part of the stable
40+
version numbers (i.e. versions >= 1.0.0) SHOULD be used for API
41+
versioning in the URL. The reason is that backward-incompatible
42+
changes are usually introduced by major changes. Therefore, the
43+
version number in the URL MUST not be used by the clients to detect
44+
the version number of the API. Use the version number returned from
45+
`GET /` instead.
46+
paths:
47+
/processing_parameters:
48+
get:
49+
summary: Additional processing parameters
50+
operationId: list-processing-parameters
51+
description: |-
52+
Lists additional custom processing parameters
53+
that a back-end offers for the different processing modes (synchronous processing, batch jobs, secondary web services).
54+
The parameters specified here can be added to the corresponding `POST` requests at the top-level of the object that is sent as the payload.
55+
All parameters SHOULD explicitly be made optional with reasonable defaults as otherwise the interoperability between the implementations decreases.
56+
tags:
57+
- Capabilities
58+
- Data Processing
59+
- Batch Jobs
60+
- Secondary Services
61+
security:
62+
- {}
63+
- Bearer: []
64+
responses:
65+
'200':
66+
description: >-
67+
An object with a list of parameters per processing mode.
68+
content:
69+
application/json:
70+
schema:
71+
description: Processing parameters per processing mode.
72+
type: object
73+
properties:
74+
create_job_parameters:
75+
$ref: '#/components/schemas/processing_create_parameters'
76+
create_service_parameters:
77+
$ref: '#/components/schemas/processing_create_parameters'
78+
create_synchronous_parameters:
79+
$ref: '#/components/schemas/processing_create_parameters'
80+
components:
81+
schemas:
82+
processing_create_parameters:
83+
title: Creation Parameters
84+
description: |-
85+
List of additional custom parameters that a back-end offers during the creation
86+
of batch jobs (`POST /jobs`) and secondary web services (`POST /services`) respectively.
87+
type: array
88+
items:
89+
$ref: '../../openapi.yaml#/components/schemas/parameter'
90+
example:
91+
- name: memory
92+
description: Maximum amount of memory that will be allocated for processing, in gigabytes.
93+
optional: true
94+
default: 32
95+
schema:
96+
type: integer
97+
minimum: 1
98+
securitySchemes:
99+
Bearer:
100+
$ref: '../../openapi.yaml#/components/securitySchemes/Bearer'

‎extensions/processing-parameters/package.json

+24
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
{
2+
"name": "@openeo/api-extension-processing-parameters",
3+
"version": "0.1.0",
4+
"author": "openEO Consortium",
5+
"license": "Apache-2.0",
6+
"description": "The openEO API specification.",
7+
"homepage": "https://openeo.org",
8+
"bugs": {
9+
"url": "https://github.com/Open-EO/openeo-api/issues"
10+
},
11+
"repository": {
12+
"type": "git",
13+
"url": "git+https://github.com/Open-EO/openeo-api.git"
14+
},
15+
"devDependencies": {
16+
"@stoplight/spectral-cli": "^6.6.0",
17+
"redoc-cli": "^0.13.21"
18+
},
19+
"scripts": {
20+
"start": "redoc-cli serve openapi.yaml --watch --options.expandResponses \"200,201,202,203,204\" --options.pathInMiddlePanel true",
21+
"build": "redoc-cli bundle openapi.yaml -o redoc.html --title \"openEO API - Processing Parameters Extension\" --cdn --options.expandResponses \"200,201,202,203,204\" --options.pathInMiddlePanel true",
22+
"test": "spectral lint openapi.yaml"
23+
}
24+
}

‎extensions/remote-process-definition/README.md

+4-3
Original file line numberDiff line numberDiff line change
@@ -13,9 +13,10 @@ This extension enables user to load user-defined processes that are hosted exter
1313
The openEO API defines the `namespace` property in a process node of a process graph as follows:
1414

1515
> The following options are predefined by the openEO API, but additional namespaces may be introduced by back-ends or in a future version of the API.
16-
> * `null` [...]
17-
> * `backend` [...]
18-
> * `user` [...]
16+
>
17+
> - `null` [...]
18+
> - `backend` [...]
19+
> - `user` [...]
1920
2021
This makes it possible for this extension to add additional allowed values to the `namespace` property.
2122

0 commit comments

Comments
 (0)