Aggressive String Limits Causing Avoidable SQLITE_TOOBIG?

[mharradon]

Hello - thanks for your work on this project!

I am running into an issue where SQLITE_TOOBIG is being thrown when querying large JSON strings. E.g. something of the form:

db.prepare("select json_extract(entry_data, '$.x') as x, from trace order by entry_index limit 1;").get();

The raw JSON string is on the order of 512MB, and the result string can be arbitrarily small (e.g. 1kB) and the error still occurs. I believe this is caused by this statement:

better-sqlite3/src/better_sqlite3.cpp

Line 499 in 5dae904

sqlite3_limit(db_handle, SQLITE_LIMIT_LENGTH, MAX_BUFFER_SIZE < MAX_STRING_SIZE ? MAX_BUFFER_SIZE : MAX_STRING_SIZE);

I think the intent was to prevent sqlite3 from returning strings larger than V8 can handle (0.5 GB now?), but I think this had the unintended consequence of preventing internal allocation allocation of larger strings (even if the result is small enough for V8).

It would be very helpful if this was increased or at least sqlite3_limit was exposed to be changed. I appreciate the desire to be conservative here by default as a hardening practice.

Thank you!