How to Batch Query with with Method

[yukefiona]

When using the with method for eager loading, it typically generates a MySQL query with an IN clause. If the data volume is too large, this can cause the query to exceed MySQL’s IN clause limit, leading to errors.

Why not use pagination?

One issue with pagination is that the associated data in the main query may contain many duplicate items. Using with will result in repeated queries, which can affect performance.

Why not query separately?

When there are multiple associations and multi-level relationships, it becomes cumbersome to handle.

Is there a way to use with to batch query all results?

[macropay-solutions]

You are seeking the impossible :)
We managed to handle streamed downloads by using "with" via cursor pagination in batches of 1000 or 2000 rows.
In approx 1 hour it can download ~700.000-1.000.000 rows from a mysql db with over 20 relations for each row.

To eliminate the time limitation that we imposed in a proxy gateway, the download can be done starting from browser in JS by using excel js which can STREAM data (csv) from the browser memory to disk.

If you want more details please ask, we will stop here.