Support streaming endpoints in WES for large responses

[JaeAeich]

Problem

Some WES responses can become extremely large depending on the workflow and implementation. For example, a GetRunLog response with many outputs can cause:

• High payload sizes
• Slow API responses
• Timeouts or degraded client performance

Currently, the WES spec doesn’t provide guidance on handling large responses beyond the standard synchronous REST calls.

Proposed Solution

Introduce streaming endpoints (or equivalent mechanisms such as pagination/chunked transfer) to handle large payloads in a scalable way.

Possible approaches:

• Add optional streaming endpoints (e.g., GET /runs/{run_id}/stream/{field}) for fields that can grow arbitrarily large (outputs, logs (I guess thats what TesLogs were meant to solve but all engines don't support TES but WES should be general enough)).
• Define a standard response format for streaming (e.g., newline-delimited JSON, chunked JSON, or pagination).
• Keep existing endpoints lightweight, returning only summaries or truncated versions of large fields, while streaming endpoints provide full access.

Related

See [issue #228] (discussion about output schema recommendations).

[JaeAeich]

cc: @uniqueg @vschnei @vinjana

[vinjana]

We have noticed this problem, although not for the standard end-points. We return actual logs via an extension route that used by a dashboard application. However, in the standard routes, is this problem not routinely prevented by returning URIs instead of the logs themselves? For instance, if the returned URI is that of an S3 bucket or some other server that supports streaming then the REST server would not be affected.

Furthermore, concerning the specific solution you propose: In REST, would a dedicated route (resource) for streaming be preferable, or should this better be modeled as a parameter to an existing route? Is there a best practice for REST API design?

[JaeAeich]

This is definitely an edge case, but a real one. I encountered it while running a large workflow using TES as the backend and WES as the interface. In that setup, execution and data were decoupled: every file (including engine caches) lived on S3. Combined with the large number of results the workflow produced, the metadata files further amplified the response size problem.

The issue goes beyond just “big outputs.” The larger the response body for a single endpoint, the more the client is forced to wait before rendering or processing anything. This becomes painfully apparent in a UI ([example](https://cloud-components.2.rahtiapp.fi/docs/wes/components/runs)):

• If the UI wants to display everything in one go, it must wait for the entire response.
• If the UI is designed with tabs, and the API exposes endpoints per tab, then the smaller ones can load quickly while the heavier ones can load later. Without streaming or chunking, this kind of optimization falls entirely on the client, with no real flexibility.

Outside of UI concerns, there are also backend limitations. For example, inserting such large responses into a NoSQL database like MongoDB is problematic, since a single document cannot exceed 16 MB. Larger payloads can either be rejected or even cause OOMKills.

As for solutions, you raise a good point about design choices:

• Pagination: Offload the “explosive” parts of the response (like lists) into a separate endpoint and paginate them.
• Streaming: Provide streaming endpoints. This gives implementors and clients more flexibility to optimize for their use case (e.g., progressive loading in UIs, partial ingestion in backends).

Regardless, I would assume any solution here should be more inclusive and consider a broader scope of impacts. I hope I’ve been able to illustrate some of the nuances I’ve faced and foreseen, though there are certainly more.

[patmagee]

This has already come up for us and we have started trying to move in the direction of striping information out of the RunLog where needed. The issue we encountered was the tasks are embedded directly in the response so we added the tasks series of endpoints.

I would probably want flags when getting the RunLog to turn off including things like outputs, tasks or anything else that could be large and then add bespoke endpoints.

I do not see why we would not have an endpoint that looks like the following

GET /ga4gh/wes/v1/run/<runId>/workflow_params

GET /ga4gh/wes/v1/run/<runId>/outputs

GET /ga4gh/wes/v1/run/<runId>/outputs/*key

For logs, I don't think we need a specific endpoint to stream the logs. The escape hatch we havee is tthat the WES service provides a link to ANY endpoint it controls that the user can stream the logs with.. its more flexible imo