Query method not working for large files

[jiexixi]

Documentation feedback

• Page title: Query CrateDB
• Page URL: https://crate.io/docs/python/en/latest/query.html
• Source: https://github.com/crate/crate-python/blob/master/docs/query.rst

---

[jiexixi]

Hi there, I tried to query some large files on crateDB with the above method but it threw the error message

crate.client.exceptions.ProgrammingError: CircuitBreakingException[[query] Data too large, data for [http-result] would be [645922816/616mb], which is larger than the limit of [644245094/614.3mb]]

It seems that the file has exceeded the size limit of the function. Could you please recommend a way to query the large files (around 18GB)? Thank you.

[proddata]

Hi @jiexixi

To retrieve large resultsets (i.e. larger than what would fit in the heap) you need to either use a client / driver that supports a fetch size like asyncpg https://github.com/MagicStack/asyncpg or JDBC on a protocol level or run multiple sub-queries and merge the result on the client.

[amotl]

Hi @jiexixi,

apologies for the late reply. Relating to the title of this ticket, are you transferring large files using the CrateDB BLOB interface, or are you running SQL queries that yield large result sets, so the circuit breakers kick in?

Relating to @proddata's answer, let me also share those resources with you:

• Fetching large result sets from CrateDB outlines relevant querying options.
• Increasing CrateDB's heap memory may also help.

With kind regards,
Andreas.