-
Notifications
You must be signed in to change notification settings - Fork 181
feat(arrow): Allow record batches output from read_sql #819
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
pub fn to_ptrs<'py>(&self, py: Python<'py>) -> Bound<'py, PyAny> { | ||
let ptrs = py.allow_threads( | ||
|| -> Result<(Vec<String>, Vec<Vec<(uintptr_t, uintptr_t)>>), ConnectorXPythonError> { | ||
let rbs = vec![self.0.clone()]; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is this okay or do you suggest any workarounds?
# doesn't work without `.clone()`, breaks with the following
cannot move out of `self` which is behind a shared reference
move occurs because `self.0` has type `arrow::array::RecordBatch`, which does not implement the `Copy` trait
else: | ||
raise ValueError(return_type) | ||
|
||
return df | ||
|
||
|
||
def reconstruct_arrow_rb(results) -> pa.RecordBatchReader: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
returns a pyarrow RecordBatchReader
instead of an iterator/ generator of RecordBatch
. I guess this will be useful for users who want to get the pyarrow Schema
since RecordBatchReader has it.
@wangxiaoying for your review. |
Thanks @chitralverma for the PR! I will take a look at it by the end of this week. |
Changes
new_record_batch_iter
to expose a pyarrowRecordBatchReader
on python sidekwargs
toread_sql
, users can passrecord_batch_size
to control the number of records in each record batch.unwraps
causing issuesRecordBatchReader
trait to supportSend
(helps offloadRecordBatchReader
to multi-threaded consumers like DuckDB)Usage/ Example
closes #278