Skip to content

Conversation

@davidhewitt
Copy link
Collaborator

This PR changes dictionary handling to eagerly upcast the keys to i64 size.

This allows us to avoid changing any logic but strip a layer of generics away, namely the possible dictionary keys. This results in a ~20x reduction in code size as measured by cargo llvm-lines:

before

Lines                  Copies                Function name
  -----                  ------                -------------
  6187111                209087                (TOTAL)
   657720 (10.6%, 10.6%)  12600 (6.0%,  6.0%)  datafusion_functions_json::common::zip_apply::inner::{{closure}}
   598662 (9.7%, 20.3%)    3789 (1.8%,  7.8%)  <arrow_array::array::primitive_array::PrimitiveArray<T> as core::iter::traits::collect::FromIterator<Ptr>>::from_iter
   552270 (8.9%, 29.2%)     280 (0.1%,  8.0%)  datafusion_functions_json::common::zip_apply

after

Lines                 Copies              Function name
  -----                 ------              -------------
  305725                9773                (TOTAL)
   34870 (11.4%, 11.4%)   60 (0.6%,  0.6%)  datafusion_functions_json::common::zip_apply
   25200 (8.2%, 19.6%)   480 (4.9%,  5.5%)  datafusion_functions_json::common::zip_apply::inner::{{closure}}
   24174 (7.9%, 27.6%)   153 (1.6%,  7.1%)  <arrow_array::array::primitive_array::PrimitiveArray<T> as core::iter::traits::collect::FromIterator<Ptr>>::from_iter

Copy link
Collaborator

@samuelcolvin samuelcolvin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@davidhewitt davidhewitt enabled auto-merge (squash) January 31, 2025 11:38
@codecov-commenter
Copy link

codecov-commenter commented Jan 31, 2025

Codecov Report

Attention: Patch coverage is 69.15888% with 33 lines in your changes missing coverage. Please review.

Project coverage is 83.28%. Comparing base (82521b6) to head (e025824).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
src/common.rs 69.15% 24 Missing and 9 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #66      +/-   ##
==========================================
- Coverage   83.82%   83.28%   -0.55%     
==========================================
  Files          15       15              
  Lines        1045     1089      +44     
  Branches     1045     1089      +44     
==========================================
+ Hits          876      907      +31     
- Misses        120      125       +5     
- Partials       49       57       +8     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@davidhewitt davidhewitt merged commit 73c8da2 into main Jan 31, 2025
7 checks passed
@davidhewitt davidhewitt deleted the dh/generic-bloat branch January 31, 2025 11:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants