Skip to content

Commit d795cf0

Browse files
authored
Merge pull request #165 from NOAA-GSL/staging
Ingestion via structured JSON from modern APIs
2 parents 9ffcdbd + 3485e7e commit d795cf0

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

65 files changed

+7620
-157
lines changed

.github/workflows/codeql.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,9 @@ name: CodeQL
33

44
on:
55
push:
6-
branches: [ main ]
6+
branches: [ main, staging ]
77
pull_request:
8-
branches: [ main, master ]
8+
branches: [ main, master, staging ]
99
workflow_dispatch:
1010
schedule:
1111
- cron: '0 6 * * 1'

.github/workflows/manual-release.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -182,6 +182,7 @@ jobs:
182182
file: ${{ matrix.dockerfile }}
183183
platforms: linux/amd64,linux/arm64
184184
push: true
185+
no-cache: true # force rebuild for release images
185186
build-args: |
186187
ZYRA_VERSION=${{ inputs.version }}
187188
${{ matrix.name == 'zyra-scheduler' && 'ZYRA_EXTRAS=connectors' || '' }}

AGENTS.md

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -106,11 +106,27 @@ Documentation Sources
106106
- Workflow Stages: https://github.com/NOAA-GSL/zyra/wiki/Workflow-Stages
107107
- Stage Examples: https://github.com/NOAA-GSL/zyra/wiki/Stage-Examples
108108
- Install & Extras: https://github.com/NOAA-GSL/zyra/wiki/Install-Extras
109-
- Pipeline Patterns: https://github.com/NOAA-GSL/zyra/wiki/Pipeline-Patterns
109+
- Pipeline Patterns: https://github.com/NOAA-GSL/zyra/wiki/Pipeline-Patterns
110110

111111
Notes
112112
- The `/docs/source/wiki/` content is generated via a scheduled sync from the GitHub Wiki. Do not hand-edit files in that folder unless the sync process specifically instructs otherwise. If you see a discrepancy between the repo copy and the live wiki, defer to the live wiki as the source of truth and file an issue to correct the mirror.
113113

114+
Documentation contribution guidelines (where to put updates)
115+
- Keep the top-level README concise. For detailed usage and options, prefer module-level READMEs and the wiki.
116+
- Module READMEs live next to code and should contain focused examples and flags:
117+
- Connectors (ingest/egress/discovery):
118+
- `src/zyra/connectors/ingest/README.md`
119+
- `src/zyra/connectors/egress/README.md`
120+
- `src/zyra/connectors/discovery/README.md`
121+
- Processing: `src/zyra/processing/README.md`
122+
- Visualization: `src/zyra/visualization/README.md`
123+
- Transform: `src/zyra/transform/README.md`
124+
- API service: `src/zyra/api/README.md`
125+
- MCP tools: `src/zyra/api/mcp_tools/README.md`
126+
- Sphinx includes these module READMEs under "Module READMEs" in `docs/source/index.rst`. If you add a new README, include it there.
127+
- Keep function/class docstrings current; autodoc consumes them for the API reference pages.
128+
- Use the GitHub Wiki for narrative guides and high-level design. The mirrored copy under `docs/source/wiki/` is not edited directly.
129+
114130
When In Doubt: Decision Flow
115131

116132
- CLI usage or flags

README.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,14 @@ zyra process convert-format sample.grib2 netcdf --stdout > sample.nc
3434
zyra visualize heatmap --input sample.nc --var VAR --output plot.png
3535
```
3636

37+
### More Examples per Module
38+
39+
See module-level READMEs under `src/zyra/` for focused examples and options:
40+
- Connectors/Ingest: `src/zyra/connectors/ingest/README.md`
41+
- Processing: `src/zyra/processing/README.md`
42+
- API (service): `src/zyra/api/README.md`
43+
- MCP Tools: `src/zyra/api/mcp_tools/README.md`
44+
3745
## Learn More (Wiki)
3846
- Getting Started: https://github.com/NOAA-GSL/zyra/wiki/Getting-Started-with-Zyra
3947
- Workflow Stages: https://github.com/NOAA-GSL/zyra/wiki/Workflow-Stages

docs/source/index.rst

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,11 +43,26 @@ Contents
4343
api/zyra.cli
4444
api/zyra.transform
4545

46+
.. toctree::
47+
:maxdepth: 1
48+
:caption: Module READMEs
49+
50+
../../src/zyra/README
51+
../../src/zyra/connectors/ingest/README
52+
../../src/zyra/connectors/egress/README
53+
../../src/zyra/connectors/discovery/README
54+
../../src/zyra/processing/README
55+
../../src/zyra/api/README
56+
../../src/zyra/api/mcp_tools/README
57+
../../src/zyra/visualization/README
58+
../../src/zyra/transform/README
59+
4660
.. toctree::
4761
:maxdepth: 2
4862
:caption: Guides
4963

5064
discovery
65+
openapi
5166
stages_overview
5267
domain_apis
5368
mcp

docs/source/openapi.rst

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
OpenAPI-Aided Help and Validation
2+
=================================
3+
4+
Overview
5+
--------
6+
7+
Zyra can use an API's OpenAPI 3.0 specification to provide request hints and
8+
perform lightweight validation before issuing requests.
9+
10+
CLI (``zyra acquire api``)
11+
--------------------------
12+
13+
- ``--openapi-help``
14+
Fetches the service's OpenAPI spec and prints a summary for the resolved
15+
operation (required/optional query and header parameters, requestBody content types,
16+
and best-effort response content types). This does not send the request.
17+
18+
- ``--openapi-validate``
19+
Validates the provided ``--header``, ``--params``, and ``--data`` (when
20+
present) against the spec. Prints issues to stderr.
21+
22+
- ``--openapi-strict``
23+
With ``--openapi-validate``, exits non-zero when issues are found.
24+
25+
API (``POST /v1/acquire/api``)
26+
------------------------------
27+
28+
- ``openapi_validate``: boolean. When true, validates the request before issuing it.
29+
Returns HTTP 400 with ``{"errors": [...]}`` on issues.
30+
- ``openapi_strict``: boolean. Defaults to true; controls whether issues are fatal.
31+
32+
What is validated
33+
-----------------
34+
35+
- Required query and header parameters.
36+
- Required ``requestBody`` presence.
37+
- ``requestBody`` Content-Type compatibility.
38+
- Simple parameter checks when values are present:
39+
- ``enum`` membership
40+
- Basic ``type`` checks (string/integer/number/boolean)
41+
- Optional JSON Schema validation for ``application/json`` requestBody when
42+
``jsonschema`` is installed in the environment.
43+
44+
Notes
45+
-----
46+
47+
- Zyra uses a best-effort resolver for operations, matching the longest
48+
templated path (e.g., ``/v1/items/{id}``) with segment-aware scoring.
49+
- When the OpenAPI spec is unavailable, help and validation are skipped.
50+
51+
Examples
52+
--------
53+
54+
Example OpenAPI snippet
55+
~~~~~~~~~~~~~~~~~~~~~~~
56+
57+
.. code-block:: json
58+
59+
{
60+
"openapi": "3.0.0",
61+
"paths": {
62+
"/v1/items": {
63+
"get": {
64+
"parameters": [
65+
{"in": "query", "name": "q", "required": true, "schema": {"type": "string"}},
66+
{"in": "query", "name": "limit", "schema": {"type": "integer"}}
67+
],
68+
"responses": {
69+
"200": {"content": {"application/json": {}}}
70+
}
71+
}
72+
},
73+
"/v1/items/{id}": {
74+
"get": {
75+
"parameters": [
76+
{"in": "path", "name": "id", "required": true, "schema": {"type": "string"}}
77+
],
78+
"responses": {
79+
"200": {"content": {"application/json": {}}}
80+
}
81+
}
82+
}
83+
}
84+
}
85+
86+
CLI
87+
~~~
88+
89+
.. code-block:: bash
90+
91+
# Print required params for GET /v1/items
92+
zyra acquire api \
93+
--url https://api.example/v1/items \
94+
--openapi-help
95+
96+
# Validate current flags (warn-only)
97+
zyra acquire api \
98+
--url https://api.example/v1/items \
99+
--params q=wind \
100+
--openapi-validate
101+
102+
# Validate strictly (exit non-zero on issues)
103+
zyra acquire api \
104+
--url https://api.example/v1/items \
105+
--openapi-validate --openapi-strict
106+
107+
HTTP
108+
~~~~
109+
110+
.. code-block:: bash
111+
112+
# Validate via API (400 on issues)
113+
curl -sS -X POST http://localhost:8000/v1/acquire/api \
114+
-H 'Content-Type: application/json' \
115+
-d '{"url": "https://api.example/v1/items", "openapi_validate": true}'
116+

ingress/configmap-env.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,4 +15,4 @@ data:
1515
# Optional: tune results cleanup/TTL (seconds)
1616
# RESULTS_TTL_SECONDS: "86400"
1717
# RESULTS_CLEAN_INTERVAL_SECONDS: "3600"
18-
18+
API_ROOT_PATH: "/zyra"

pyproject.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
[tool.poetry]
22
name = "zyra"
3-
version = "0.1.33"
3+
version = "0.1.34"
44
description = "A tool to ingest data from various sources and formats, create imagery or video based on that data, and send the results to various locations for dissemination."
55
authors = ["Eric Hackathorn <eric.j.hackathorn@noaa.gov>"]
66
include = [
Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
[
2+
"/",
3+
"/health",
4+
"/llm/test",
5+
"/ready",
6+
"/v1/acquire",
7+
"/v1/acquire/api",
8+
"/v1/assets",
9+
"/v1/cli/commands",
10+
"/v1/cli/examples",
11+
"/v1/cli/run",
12+
"/v1/commands",
13+
"/v1/decide",
14+
"/v1/decimate",
15+
"/v1/disseminate",
16+
"/v1/enrich",
17+
"/v1/export",
18+
"/v1/health",
19+
"/v1/import",
20+
"/v1/jobs/{job_id}",
21+
"/v1/jobs/{job_id}/download",
22+
"/v1/jobs/{job_id}/manifest",
23+
"/v1/llm/test",
24+
"/v1/mcp",
25+
"/v1/mcp/progress/{job_id}",
26+
"/v1/narrate",
27+
"/v1/optimize",
28+
"/v1/presets/limitless/audio",
29+
"/v1/process",
30+
"/v1/process/api-json",
31+
"/v1/ready",
32+
"/v1/render",
33+
"/v1/search",
34+
"/v1/search/profiles",
35+
"/v1/simulate",
36+
"/v1/transform",
37+
"/v1/upload",
38+
"/v1/verify",
39+
"/v1/visualize"
40+
]
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
6da0d92837e53b443550532c1da2860ed55a68b5010d3d568a53a03fd578ba41

0 commit comments

Comments
 (0)