StreamedJsonResponse not streaming -> alternative solution

[macropay-solutions]

Feature implemented in #49873 (comment)

@pelmered
We just finished implementing this StreamedJsonResponse in our cruFd lib suite and we saw it was not streaming at all, but just builds the response in output buffer until it fills the memory if the table is big enough.

In symfony documentation they mention that you should use flush once in a while.

We did flush after each 100 rows and also we put a sleep after flushing, but no luck.
The memory issue persisted.

Analog with ob_flush().

We tested this on our demo page/website. Could there be a server setup that prevents streaming or is this an issue from symfony?

Update

We set in php.ini

output_buffering = 1024

and then the response changed to

{
    "has_more_pages": false,
    "sums": {
        "value": null
    },
    "avgs": {
        "value_avg": null
    },
    "mins": {
        "value_min": null,
        "created_at_min": null
    },
    "maxs": {
        "value_max": null,
        "created_at_max": null
    },
    "index_required_on_filtering": [
        "id",
        "parent_id",
        "client_id",
        "created_at",
        "currency",
        "value"
    ],
    "current_page": 1,
    "from": 1,
    "last_page": 1,
    "per_page": 3748918,
    "to": 3748918,
    "total": 3748918,
    "data":

Previously it was valid json but data did not contain any rows, just a "message": "Server error"

[macropay-solutions]

Locally it behaves the same. No streaming involved.

index.php:

$response = $kernel->handle(
    $request = Request::capture()
)->send();

Maybe because send is not called until the end? The send is called before the yielding starts.

The StreamedResponse seems to be flushing the data after each echo.

We tried with cUrl in parallel terminals. In one we tailed the logs while in the other we ran the call.

For StreamedResponse both terminals were showing new data async.

For StreamedJsonResponse first the yield happened in logs and when that finished then it started sending the data, so in a sync way.

For csv download (StreamedResponse) we did not use generator (yield) and no flush or ob_flush.

Locally we use:

PHP 8.2.17 (cli) (built: Mar 16 2024 01:08:19) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.2.17, Copyright (c) Zend Technologies
with Xdebug v3.3.1, Copyright (c) 2002-2023, by Derick Rethans

The only difference between the 2 situations seems to be LazyCollection (json) vs EloquentCollection (csv).

[macropay-solutions]

Acc to this jsons can't be streamed while using json_encode, which is used in this case.

[macropay-solutions]

symfony/symfony#60252

[pelmered]

Yeah, we had some problems with this as well. We didn't dig much deeper into this than you have it looks like and we ended up abandoning this.

Our plan now is to switch to Event Streams instead, but we haven't tried that yet. That looks to be pretty straight forward to handle in the front end as well by the looks of it: https://developer.mozilla.org/en-US/docs/Web/API/EventSource

[macropay-solutions]

@pelmered The event stream solves another issue mainly not the streaming of json. We will try to make this one work.

[macropay-solutions]

Good news is we made it work from the BE as stream. Now we are working on a JS example/demo to consume that stream.

[pelmered]

Great job!
Please share when you get this to work. Or, even better, update the Laravel docs with an example on how to make this work.

[macropay-solutions]

@pelmered
The Symfony class is not usable for big amounts of data.

We created a new stream response class. Will post the link to the demo page after we deploy it.

Until then you can see how the current implementation works (with a limit of 20000 rows to avoid the memory issue). This is usable instead of using total as per page via callback in the paginator but that would also need a top limit on that callback.

https://laravel-crud-wizard.com/laravel-10/laravel-lumen-crud-wizard#operations

In operations menu put in text area limit=-1 or select Streamed Json from

On submit it will open a new tab and prompt for saving a json file.

[macropay-solutions]

Calling after each yield + echo for 20000 rows:
flush (low number of handled errors due to incomplete json streamed/received)
vs
ob_flush + flush (higher number but the data gets printed faster in html)

[macropay-solutions]

@pelmered

php class with JS example code for handling the new streamed json
https://github.com/macropay-solutions/laravel-crud-wizard-free/blob/production/src/Responses/StreamedJsonResponse.php

[macropay-solutions]

@pelmered
demo for the new solution on laravel 9 with scope condition on id <= 20000 to not bloat the html (it has also max 5 min php execution time limit)
https://laravel-crud-wizard.com/laravel-9/laravel-lumen-crud-wizard#operations

This will not open a new tab with prompt for download the json but it will print it in html.

We will keep the demo on laravel 10 with the not streaming version of StreamedJsonResponse for a while to see the differences.

We published this in laravel-crud-wizard-free v 3.5.0 https://github.com/macropay-solutions/laravel-crud-wizard-free/releases/tag/3.5.0 but NOT in its demo pages.

[macropay-solutions]

Update for symfony. We gave them a demo project to reproduce the issue. symfony/symfony#60257 (comment)