Skip to content

Bulk Synchronous Standards #69

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Nov 20, 2023
Merged

Bulk Synchronous Standards #69

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
1891901
Initial draft of Bulk payload
travisgosselin Oct 25, 2023
6858fcf
Some reference updates
travisgosselin Oct 25, 2023
5f28b58
Update standards/bulk.md
travisgosselin Oct 26, 2023
5d1a769
Update standards/bulk.md
travisgosselin Oct 26, 2023
3fdd093
Bulk conventions update
travisgosselin Nov 14, 2023
405a6a1
Merge branch 'main' into feature/sync-bulk
travisgosselin Nov 14, 2023
5d7b123
A few updates
travisgosselin Nov 16, 2023
d067b88
Update transaction mode and status to be in result
travisgosselin Nov 16, 2023
b71764f
Updated for PR
travisgosselin Nov 17, 2023
412e217
Merge branch 'main' into feature/sync-bulk
travisgosselin Nov 20, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ To easily review, read and reference these standards refer to documentation publ

1. [URL Structure](standards/url-structure.md) - URLs, Resources, Hierarchies and Query Parameters.
1. [Request & Response](standards/request-response.md) - Verbs/Methods, Headers and Status Codes.
1. [Bulk Operations](standards/bulk.md) - Handling bulk operations with RESTful compromises.
1. [Naming](standards/naming.md) - General, Text, Property Names and Standard Properties.
1. [Serialization](standards/serialization.md) - Casing, Types, Quantities, Intervals and Durations.
1. [Collections](standards/collections.md) - Results Body, Pagination, Searching, Filtering and Sorting.
Expand Down
174 changes: 174 additions & 0 deletions standards/bulk.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,174 @@
# Bulk Operations

## Overview

The concept of bulk operations on RESTful endpoints is quite foreign, in the sense that it is not a common practice. However, there are some use cases where it makes sense to allow a client to perform multiple operations in a single request. For example, a client may want to create multiple entities at once, or update multiple entities at once. In these cases, it is often more efficient to allow the client to perform these operations in a single request, rather than making multiple requests. However, before adding support for bulk operations you should think twice if this feature is really needed. Often network performance is not what limits request throughput. Similarly, sending multiple in-parallel requests for standard REST endpoints can be a fine solution for some clients.

### Synchronous

Bulk operations **MUST** be synchronous when applied to an existing resource. This means that the client will receive a response to the bulk request only after all operations have been completed. This is in contrast to asynchronous operations, where the client receives a response immediately after the request is received, and then must poll for the result of the operation. Asynchronous bulk operations are applied to new resources specifically for bulk or import to enable subsequent status updates through additional endpoints.

- Bulk operations **MUST** be specific to a single resource type and NOT allow for updating multiple resource types in a single request.
- Bulk operations **MUST** be implemented as a PATCH request against a collection resource that not idempotent.
- Bulk operations **MUST** return a `200 (OK)` response code if all operations were received and a result is available for each operation. Bulk operations **MUST NOT** use status code `207 (Multi-Status)` response code as this incurs other implications for the response schema in relationship to WebDAV. System level errors may still result in a `500 (Internal Server Error)` response code where the request could not be processed or was prevented from trying operations.
- Bulk operations **MUST** accept a constrained number of operations in the request body that is indicated in the documentation of the endpoint. By default this value **MAY** be 100 operations, but should be adjusted according to the needs of the endpoint and the entity-size. Requests beyond the limit **MUST** return a `400` response code and standard [error format body](errors.md) similar to:
```json
// RESPONSE
HTTP/1.1 400
Content-Type: application/problem+json
{
"title": "Invalid Data",
"status": 400,
"detail": "Operations collection may only contain a maximum of '100' actions per request.",
"instance": "/articles",
"requestId": "b6d9a290-9f20-465b-bcd3-4a5166eeb3d7"
}
```

#### Request

- Bulk operations **MUST** include a request body schema
```json
{
"transaction": "ATOMIC" | "ISOLATED", // optional indication of transactionality. default is "ISOLATED"
"operations": [
{
"operationId": "string" | null, // optional consumer generated id to associate with the operation for comparison to result.
"action": "CREATE" | "UPDATE" | "CREATE_UPDATE" | "DELETE", // indicate intent of operation (can use subset, but do not extend)
"ifMatch": "string" | null, // if-match is an optional ETag that can be passed for optimistic concurrency
"entity": {
... MATCHING ENTITY... // must match entity schema resource from the collection
} // schema on entity is static and not dynamic
},
]
}
```

- Bulk operation requests **MAY** be designated as `ATOMIC` or `ISOLATED` via the `transaction` field.
- The default transaction type **MUST** be `ISOLATED`.
- The transaction type **MAY** be optionally left out of the request schema in implementation.
- `ATOMIC` transactions will either succeed or fail together.
- `ISOLATED` transactions will allow for individual operations to succeed or fail independently.
- `operationId` **MAY** be used to associate the operation in the request with the resulting operation in the response. This is useful for tracking the outcome of each operation in the response where it might be ambiguous to reference via `entityId`.
- `action` enumeration **MUST NOT** be extended, but **MAY** be a subset of the enumeration values where not all actions are required.
- `ifMatch` **MUST** be supported in the request schema if ETags are used for other RESTful operations on the same resource.
- Operation collections in a request **INCLUDE** include entities of the same `id` only once. If the collection contains multiple entities with the same `id`, then the request **MUST** return a `400 Bad Request` status code and error body.

#### Response

- Bulk operations **MUST** include a response body schema as follows, which is not extensible:
```json
{
"status": "SUCCEEDED" | "FAILED" | "PARTIAL", // overall status of bulk operation, partial indicating
"operations": [ // there are some operations that failed and succeeded
{
"operationId": "string", // matching operation id from the request body if provided, otherwise use index value as a string
"action": "CREATE" | "UPDATE" | "CREATE_UPDATE" | "DELETE", // repeat action type
"entityId": "string" | null, // the associated id of the entity, if available
"entityRef": "sps-ref" | null, // the associated sps-ref URN entity, if applicable
"status": "SUCCEEDED" | "FAILED", // status of individual operation
"detail": { // optional detail information about the operation, like error details
"message": "string", // if detail object is provided, it must include a message
"code": "string" | null, // codes can be similar to those used in Error Response, or custom for other purposes, optional in schema
"field": "string" | null, // field indicates an associated field in the entity to highlight, optional in schema
"value": "string" | null // the value of the associated field highlighted in the detail, optional in schema
}
}
]
}
```

- `status` enumeration **MUST NOT** be extended or filtered.
- `operations` collection **SHOULD** be in the same order as the request body operations.
- `operationId` in the response **MUST** match the `operationId` of the request body where provided, otherwise it should fallback to the index of the operation in the request body as a string value.
- `entityId` and `entityRef` should both be used to identify the primary ID or an associated [ref](naming.md) value for the entity. These are optional and can be left out if not applicable.
- `detail` field **MAY** be included as a complex object following the identified schema. If it is include, then it **MUST** include a `message` field. It **MUST NOT** be extended. The additional fields around `code`, `field`, and `value` are optional and **MAY** be left out if not applicable to your resource

#### Example

The following example demonstrates various operations in a bulk request and response:

```json
// REQUEST
PATCH /articles
Content-Type: application/json
{
"operations": [
{
"action": "CREATE_UPDATE",
"ifMatch": "33a64df551425fcc55e4d42a148795d9f25f89d4",
"entity": {
"id": "bfd8f0c0-be67-4f81-bf82-e55e552609f4",
"name": "my name",
"description": "my description"
}
},
{
"action": "CREATE",
"operationId": "my-unique-id-or-uuid",
"entity": {
"id": null,
"name": "my name",
"description": "my description"
}
},
{
"action": "DELETE",
"ifMatch": "44a64df551425fcc55e4d42a148795d9f25f89c5",
"entity": {
"id": "d9bd5d91-fc25-4410-ae42-c8f631e8e9ff",
"name": null,
"description": null
}
}
]
}

// RESPONSE
200 OK
Content-Type: application/json
{
"status": "PARTIAL",
"operations": [
{
"operationId": "0",
"action": "CREATE_UPDATE",
"entityId": "bfd8f0c0-be67-4f81-bf82-e55e552609f4",
"entityRef": "sps:thing:bfd8f0c0-be67-4f81-bf82-e55e552609f4",
"status": "SUCCEEDED",
"detail": {
"message": "Article was updated.",
"code": null,
"field": null,
"value": null
}
},
{
"operationId": "my-unique-id-or-uuid",
"action": "CREATE",
"entityId": null,
"entityRef": null,
"status": "FAILED",
"detail": {
"message": "Could not create article, since the name already exists.",
"code": "UNIQUE_NAME_VIOLATION",
"field": "name",
"value": "my name"
}
},
{
"operationId": "2",
"action": "DELETE",
"entityId": "d9bd5d91-fc25-4410-ae42-c8f631e8e9ff",
"entityRef": "sps:thing:d9bd5d91-fc25-4410-ae42-c8f631e8e9ff",
"status": "SUCCEEDED",
"detail": null
}
]
}
```

```note
**PATCH REQUEST ATOMICITY**
While the [HTTP PATCH RFC](https://datatracker.ietf.org/doc/html/rfc5789) requires full atomicity in application of the patched document, this is not possible in the case of bulk operations. In the case of bulk operations, the request body is a collection of resources which means that the request body is not a single document. While not a perfect interpretation of PATCH execution, the boundaries of bulk operations require some compromise to benefit developer experience and consistency.
```
6 changes: 3 additions & 3 deletions standards/request-response.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ When responding to API requests, the following status code ranges **MUST** be us
- Success **MUST** be reported with a status code in the `2xx` range.
- Reason phrases **MUST NOT** be modified or customized. The default reason phrases deliver an industry-standard experience for API consumers. Use the response payload as necessary to communicate further reasoning.
- HTTP status codes in the `2xx` range **MUST** be returned only if the complete code execution path is successful. There is no such thing as partial success.
- Bulk request operations **MUST** return a 200 status code with a response body indicating failures as part of the payload for each processed entity, unless all processing failures due to a system issue, in which case it's appropriate to issue a standard `5xx` error message.
- [Bulk synchronous](bulkd.md) request operations **MUST** return a `200`` status code with a response body indicating failures as part of the payload for each processed entity.
- Failures **MUST** be reported in the `4xx` or `5xx` range. This is true for both system errors and application errors.
- All status codes used in the `4xx` or `5xx` range **MUST** return standardized error responses as outlined under [Errors](errors.md).
- A server returning a status code in the `2xx` range **MUST NOT** return any error models defined in [Errors](errors.md), or any HTTP status code, as part of the response body.
Expand Down Expand Up @@ -843,7 +843,7 @@ DELETE /articles/3
204 OK
```

The [request body of a `DELETE` is not allowed](https://stackoverflow.com/questions/299628/is-an-entity-body-allowed-for-an-http-delete-request). At times an individual may intend to perform a bulk `DELETE` action and want to provide a request body or need information about the response of a `DELETE`. You will find that in this case, a `PATCH` request against the collection is an appropriate alternative.
The [request body of a `DELETE` is not allowed](https://stackoverflow.com/questions/299628/is-an-entity-body-allowed-for-an-http-delete-request). At times an individual may intend to perform a [bulk `DELETE`](bulk.md) action and want to provide a request body or need information about the response of a `DELETE`. You will find that in this case, a `PATCH` request against the collection is an appropriate alternative.

```
// REQUEST
Expand Down Expand Up @@ -889,7 +889,7 @@ Content-Type: application/json
```

```note
**Consider**: The JSON `PATCH` format is a standardized format and schema for defining a series of patch-related updates to a single JSON document, or resource: [http://jsonpatch.com](http://jsonpatch.com/).
**Consider**: For updating multiple entities in bulk on a given resource refer to [bulk operations](bulk.md).
```

### HEAD
Expand Down