Skip to content

Commit 79136f1

Browse files
aaronfrankeaaronfranke
committed
Add extension for binary buffers, buffer views, and binary file format
1 parent a15f121 commit 79136f1

File tree

7 files changed

+401
-2
lines changed

7 files changed

+401
-2
lines changed

‎spec/v0.6/extensions/binary-data/accessor.schema.json‎

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
{
2+
"$schema": "https://json-schema.org/draft/2020-12/schema",
3+
"$id": "accessor.schema.json",
4+
"title": "@ocif/canvas/accessor",
5+
"type": "object",
6+
"description": "An accessor is a typed interpretation of the data in a buffer view. As an analogy, if a buffer is a disk and a buffer view is a partition, then an accessor is a file system.",
7+
"properties": {
8+
"id": {
9+
"type": "string",
10+
"description": "A unique identifier for the accessor."
11+
},
12+
"bufferView": {
13+
"type": "string",
14+
"description": "The ID of the buffer view that contains the data for this accessor. This is a reference to a buffer view in the OCIF file's document-level `bufferViews` array."
15+
},
16+
"primitiveType": {
17+
"type": "string",
18+
"description": "The primitive data type used for each primitive component in the accessor.",
19+
"anyOf": [
20+
{
21+
"const": "float8",
22+
"description": "8-bit or 1-byte IEEE-like quarter-precision floating point number with 1 sign bit, 4 exponent bits, 3 mantissa bits, max finite value 240."
23+
},
24+
{
25+
"const": "float16",
26+
"description": "16-bit or 2-byte IEEE 754 half-precision floating point number."
27+
},
28+
{
29+
"const": "float32",
30+
"description": "32-bit or 4-byte IEEE 754 single-precision floating point number."
31+
},
32+
{
33+
"const": "float64",
34+
"description": "64-bit or 8-byte IEEE 754 double-precision floating point number."
35+
},
36+
{
37+
"const": "float128",
38+
"description": "128-bit or 16-byte IEEE 754 quadruple-precision floating point number."
39+
},
40+
{
41+
"const": "int8",
42+
"description": "8-bit or 1-byte two's complement signed integer."
43+
},
44+
{
45+
"const": "int16",
46+
"description": "16-bit or 2-byte two's complement signed integer."
47+
},
48+
{
49+
"const": "int32",
50+
"description": "32-bit or 4-byte two's complement signed integer."
51+
},
52+
{
53+
"const": "int64",
54+
"description": "64-bit or 8-byte two's complement signed integer."
55+
},
56+
{
57+
"const": "int128",
58+
"description": "128-bit or 16-byte two's complement signed integer."
59+
},
60+
{
61+
"const": "uint8",
62+
"description": "8-bit or 1-byte unsigned integer."
63+
},
64+
{
65+
"const": "uint16",
66+
"description": "16-bit or 2-byte unsigned integer."
67+
},
68+
{
69+
"const": "uint32",
70+
"description": "32-bit or 4-byte unsigned integer."
71+
},
72+
{
73+
"const": "uint64",
74+
"description": "64-bit or 8-byte unsigned integer."
75+
},
76+
{
77+
"const": "uint128",
78+
"description": "128-bit or 16-byte unsigned integer."
79+
},
80+
{
81+
"type": "string",
82+
"description": "An extension-defined primitive type."
83+
}
84+
]
85+
},
86+
"vectorSize": {
87+
"type": "integer",
88+
"description": "The number of primitives in each element of the accessor. For scalars this is 1, for 2D vectors this is 2, for 3D vectors this is 3, for 4D vectors this is 4, and so on. Matrices can be encoded as many-dimensional vectors, such as a 4x4 matrix with this property set to 16. This number MUST be a positive integer. If not specified, the default is 1, which means each primitive is a scalar element.",
89+
"minimum": 1,
90+
"default": 1
91+
}
92+
},
93+
"required": ["id", "bufferView", "primitiveType"]
94+
}

‎spec/v0.6/extensions/binary-data/binary-data.md‎

Lines changed: 176 additions & 0 deletions
Large diffs are not rendered by default.

‎spec/v0.6/extensions/binary-data/binary-file-format.md‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
# OCIF Binary File Format
2+
3+
OCIF files may be stored in a JSON-based text format (`.ocif`) or a binary format (`.ocb`, "OCIF Binary"). With the text format, binary blobs of data may either be base64-encoded within the JSON, or referenced as external files, as defined in [OCIF Binary Data Extension](binary-data.md). The binary format is a more compact representation of the same data within a self-contained file, which appends binary blobs of data after the end of the JSON.
4+
5+
The binary format begins with a 16-byte file header, which contains the following fields:
6+
7+
- A 4-byte magic number.
8+
- This MUST be equal to the byte sequence `0x4F 0x43 0x49 0x46`, or ASCII string "OCIF".
9+
- When interpreted as a little-endian unsigned 32-bit integer, this is `0x4649434F`.
10+
- A 4-byte version number.
11+
- This MUST be equal to the byte sequence `0x00 0x00 0x00 0x00`, or zero.
12+
- This value is only 0 for the draft version of the specification. The final version will have a different value.
13+
- A 8-byte size number.
14+
- This MUST be equal to the total size in bytes of the entire file, including the file header, all chunks, all JSON data, and all binary blobs of data.
15+
- This value is a little-endian unsigned 64-bit integer, with the most significant bit set to zero. The maximum file size of a binary OCIF file is 2^63 - 1 bytes.
16+
17+
After the file header, the file consists of a series of one or more chunks. Each chunk begins with its own 16-byte chunk header, with a similar format to the file header:
18+
19+
- A 4-byte chunk type indicator.
20+
- In the base specification, this MUST be one of the following:
21+
- The byte sequence `0x4A 0x53 0x4F 0x4E`, the ASCII string "JSON". This indicates the chunk contains JSON data.
22+
- When interpreted as a little-endian unsigned 32-bit integer, this is `0x4E4F534A`.
23+
- JSON chunks MUST be UTF-8 encoded without a BOM, MUST NOT contain control characters `0x7F` or `0x00` through `0x1F` except for optionally tab `0x09` and line feed `0x0A`, and MUST be a valid JSON object. These requirements also apply to the text format.
24+
- The byte sequence `0x42 0x4C 0x4F 0x42`, the ASCII string "BLOB". This indicates the chunk contains binary blob data, usually the data of a buffer.
25+
- When interpreted as a little-endian unsigned 32-bit integer, this is `0x424F4C42`.
26+
- Implementations MAY define additional chunk types, but this is usually not needed. The byte sequence selected SHOULD be a somewhat-human-readable magic sequence of printable ASCII characters, but may be any value. Note: This does not need to match the magic number used by the data format itself, if any.
27+
- A 4-byte chunk compression format indicator.
28+
- In the base specification, this MUST be one of the following:
29+
- The byte sequence `0x00 0x00 0x00 0x00`, or zero. This indicates the chunk is not compressed.
30+
- The byte sequence `0x5A 0x73 0x74 0x64`, the ASCII string "Zstd". This indicates the chunk is compressed using the Zstandard compression format.
31+
- When interpreted as a little-endian unsigned 32-bit integer, this is `0x6474735A`.
32+
- The chunk data MUST also include Zstd's own magic number `0x28 0xB5 0x2F 0xFD` at the start of the data, it cannot be omitted.
33+
- The byte sequence `0xFF 0xFF 0xFF 0xFF` is reserved as an error value, and MUST NOT be used as a compression format indicator, or else the file is invalid.
34+
- If the chunk represents the data of a buffer, this MUST match the `"compression"` property in the OCIF JSON data for that buffer, with the property undefined for uncompressed data.
35+
- Implementations MUST support the uncompressed format. Implementations MAY choose to implement any or none of the other compression formats, refusing to load such files.
36+
- Implementations MAY define additional compression formats. The byte sequence selected SHOULD be a somewhat-human-readable magic sequence of printable ASCII characters, but may be any value. Note: This does not need to match the magic number used by the compression format itself.
37+
- A 8-byte chunk data size number.
38+
- This MUST be equal to the size in bytes of the chunk data, excluding the chunk header, and excluding any padding after the chunk data.
39+
- This value is a little-endian unsigned 64-bit integer, with the most significant bit set to zero. The maximum chunk data size is 2^63 - 33 bytes (an additional 32 bytes are subtracted for the file and chunk headers).
40+
- If the chunk is compressed, this value is the size of the compressed data, not the uncompressed data. However, the `"byteLength"` field in the JSON data refers to the uncompressed size of the data.
41+
- The data in each chunk immediately follows the chunk header, and is of the size in bytes indicated by the chunk data size number.
42+
43+
Every chunk header MUST be aligned to a 16-byte boundary. This means that whenever a chunk has another chunk after it, the chunk on the left MUST have padding placed after the data (not included in the chunk data size) to the next 16-byte boundary (if already at the boundary, there is no padding). The final chunk in the file does not need padding after it. Padding is usually null `0x00` bytes for binary blobs or compressed chunks, but space `0x20` characters SHOULD be used for padding uncompressed JSON data. The 8-byte chunk data size number MUST NOT include this padding, it only includes the used bytes of the chunk data.
44+
45+
The OCIF file header magic number, chunk header type indicators, and chunk header compression format indicators are all [FourCC](https://en.wikipedia.org/wiki/FourCC) codes. This allows for easy identification of the headers if a human inspects the file in a hex editor. Implementations MAY choose to display these values to the user as plain text, such as by showing the compression format indicator in an error message when the compression format is unsupported.
46+
47+
The first chunk in the file MUST be a JSON chunk containing the OCIF JSON data that conforms to the main OCIF schema `schema.json`. If the file has binary blob chunks containing OCIF buffers, they MUST be the following chunks in the file. The OCIF JSON data's buffers array refer to these chunks, such that the buffer at index 0 uses the second chunk if `"uri"` is not defined in that buffer, the buffer at index 1 uses the third chunk if `"uri"` is not defined in that buffer, and so on. This means the `"uri"` property takes precedence over the chunk, so it should be absent in order to use the chunk's bytes. The OCIF JSON buffer's `"byteLength"` always refers to the uncompressed size of the data, but the chunk's data size refers to the compressed size of the data if the chunk is compressed.
48+
49+
All chunks used by buffers MUST immediately follow the first JSON chunk, before any custom chunks. All buffers using chunks MUST be contiguous from the start, so for example, it is not allowed for buffer 0 to use a URI and buffer 1 to use a chunk. The first buffer containing `"uri"` indicates that no buffers beyond that point may use chunks, and any buffers after that point MUST use URIs. The behavior of additional custom chunks not used by buffers is undefined, and may be used for any purpose.
50+
51+
Binary OCIF files smaller than 32 bytes are invalid, because that is the minimum size of the file header and the first chunk header. The minimum possible size of a valid uncompressed OCIF binary file is 57 bytes, which is 32 bytes for the headers and 25 bytes for the minimal JSON data of `{"asset":{"dimension":4}}`. However, such a file contains no data, and is not useful.
52+
53+
Following all of the above rules, the data layout of a OCIF binary file can be summarized as follows:
54+
55+
| Offset in bytes | Size in bytes | Description | Valid values |
56+
| --------------- | ------------- | ------------------------------------------------- | ------------------------------------------------ |
57+
| 0 | 4 | The binary OCIF file header's magic number. | Constant `OCIF` or `0x4F 0x43 0x49 0x46` |
58+
| 4 | 4 | The binary OCIF file header's version number. | Constant based on the spec version |
59+
| 8 | 8 | The binary OCIF file size in bytes. | Between 32 and 2^63 - 1 |
60+
| 16 | 4 | The first chunk's chunk type. | Constant `JSON` or `0x4A 0x53 0x4F 0x4E` |
61+
| 20 | 4 | The first chunk's compression format. | Constant `0x00000000` for uncompressed |
62+
| 24 | 8 | The first chunk's size in bytes. | Between 0 and 2^63 - 33 |
63+
| 32 | N | The first chunk's data, the OCIF JSON data. | UTF-8 encoded JSON excluding control characters |
64+
| 32 + N | P1 | (optional) Padding if a second chunk exists. | 0 to 15 null bytes or spaces to 16-byte boundary |
65+
| 32 + N + P1 | 4 | (optional) The second chunk's type. | Constant `BLOB` or `0x42 0x4C 0x4F 0x42` |
66+
| 36 + N + P1 | 4 | (optional) The second chunk's compression format. | Constant `0x00000000` for uncompressed |
67+
| 40 + N + P1 | 8 | (optional) The second chunk's size in bytes. | Between 0 and 2^63 - (48 + N + P1) |
68+
| 48 + N + P1 | M | (optional) The second chunk's data. | Binary blob data |
69+
| 48 + N + M + P1 | P2 | (optional) Padding if a third chunk exists. | 0 to 15 null bytes or spaces to 16-byte boundary |
70+
71+
More chunks may follow the second chunk, including additional BLOB chunks for buffers, or any other chunk type.
72+
73+
For the file size number, and the size of all chunks, the most significant bit is reserved for future use, and MUST be set to zero. Implementations MUST reject files or chunks where this bit is set to one. This allows for future expansion of the specification to support larger files, such as 128-bit file sizes or beyond, without breaking compatibility, in a similar manner to how UTF-8 extends ASCII.

‎spec/v0.6/extensions/binary-data/buffer-view.schema.json‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
{
2+
"$schema": "https://json-schema.org/draft/2020-12/schema",
3+
"$id": "buffer-view.schema.json",
4+
"title": "@ocif/canvas/buffer-view",
5+
"type": "object",
6+
"description": "A buffer view is a slice of a buffer, defining a region of the buffer that stores data. As an analogy, if a buffer is a disk and an accessor is a file system, then a buffer view is a partition.",
7+
"properties": {
8+
"id": {
9+
"type": "string",
10+
"description": "A unique identifier for the buffer view."
11+
},
12+
"buffer": {
13+
"type": "integer",
14+
"description": "The index of the buffer that contains the data for this buffer view. This is a reference to a buffer in the OCIF file's document-level buffers array. Buffers are only intended to be referenced by buffer views, therefore they are referenced by index rather than by ID to discourage direct access in other OCIF contexts.",
15+
"minimum": 0
16+
},
17+
"byteLength": {
18+
"type": "integer",
19+
"description": "The length of the buffer view in bytes. The declared byte offset plus the declared byte length MUST NOT exceed the buffer's byte length. This property is required.",
20+
"minimum": 0
21+
},
22+
"byteOffset": {
23+
"type": "integer",
24+
"description": "The start offset of the buffer view in bytes. This is relative to the start of the buffer. For example, if the byte offset is 100, then byte 0 of the buffer view is byte 100 of the buffer. The declared byte offset plus the declared byte length MUST NOT exceed the buffer's byte length. This property is optional and defaults to 0.",
25+
"minimum": 0,
26+
"default": 0
27+
}
28+
},
29+
"required": ["id", "buffer", "byteLength"]
30+
}

‎spec/v0.6/extensions/binary-data/buffer.schema.json‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
{
2+
"$schema": "https://json-schema.org/draft/2020-12/schema",
3+
"$id": "buffer.schema.json",
4+
"title": "@ocif/canvas/buffer",
5+
"type": "object",
6+
"description": "A buffer is a chunk of raw binary data. Buffer data may be stored in one of three ways: in an external file, in a base64-encoded string, or in the case of binary OCIF files (`.ocb`), the buffer at index 0 is used for the data chunk after the JSON at the end of the file. As an analogy, if a buffer view is a partition and an accessor is a file system, then a buffer is the whole disk.",
7+
"properties": {
8+
"byteLength": {
9+
"type": "integer",
10+
"description": "The length of the buffer in bytes. MUST NOT be negative. The actual size of the buffer data MAY be a few bytes larger than the declared length, in which case the extra bytes are unused, but MUST NOT be smaller. If the data is compressed, this refers to the uncompressed size. This property is required.",
11+
"minimum": 0
12+
},
13+
"compression": {
14+
"type": "string",
15+
"description": "The compression algorithm used for the buffer data. MUST be a 4-byte string matching the binary compression format indicator magic number. For example, `Zstd` for Zstandard compression. If not specified, the buffer is uncompressed."
16+
},
17+
"uri": {
18+
"type": "string",
19+
"description": "The URI of the buffer. This may either be a relative URI to an external file, or a base64-encoded string with the prefix `data:mimetype;base64,` where mimetype is the MIME type of the data, usually `application/octet-stream`, so the header would be `data:application/octet-stream;base64,` followed by the base64-encoded data. For a binary OCIF file, buffers with data stored in chunks MUST NOT have a URI defined."
20+
}
21+
},
22+
"required": ["byteLength"]
23+
}

‎spec/v0.6/schema.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
{
22
"$schema": "https://json-schema.org/draft/2020-12/schema",
3-
"title": "OCIF core 0.5",
3+
"title": "OCIF core 0.6",
44
"description": "The schema for the Open Canvas Interchange Format (OCIF) Core document structure.",
55
"type": "object",
66
"properties": {

‎spec/v0.6/spec.md‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1619,12 +1619,15 @@ It MAY NOT start with a hash-mark (#).
16191619

16201620
It may not contain control characters like a null byte (00), form feed, carriage return, backspace, and similar characters. In general, OCIF IDs should be [valid HTML IDs](https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Global_attributes/id) and if possible even [valid CSS identifiers](https://www.w3.org/TR/CSS2/syndata.html#value-def-identifier): "In CSS, identifiers (including element names, classes, and IDs in selectors) can contain only the characters [a-zA-Z0-9] and ISO 10646 characters U+00A0 and higher, plus the hyphen (-) and the underscore (_); they cannot start with a digit, two hyphens, or a hyphen followed by a digit."
16211621

1622-
16231622
It must be unique among all IDs used in an OCIF document.
16241623
The ID space is shared among nodes, relations, and resources.
16251624

16261625
NOTE: An OCIF file itself can be used as a resource representation. Thus, a node can show a (then nested) other OCIF file. The ID uniqueness applies only within each OCIF file, not across document boundaries.
16271626

1627+
## Integer
1628+
1629+
A `number` that represents an integer value, while JSON numbers are normally all floating-point values.
1630+
16281631
## MIME Type
16291632

16301633
A `string` that represents the _MIME Type_ for a resource.

0 commit comments

Comments
 (0)