python-v1.0.1

Bug Fixes

• fix: constraint parsing, roundtripping by @ion-elgreco in #3503

Other Changes

• docs: update link to df by @rluvaton in #3489
• chore: update runner by @ion-elgreco in #3494
• ci: improve coverage collection by @roeap in #3497
• chore: prepare for the next python release by @rtyler in #3498
• chore!: remove get_earliest_version by @roeap in #3499
• refactor!: have DeltaTable::version return an Option by @roeap in #3500

New Contributors

• @rluvaton made their first contribution in #3489

Full Changelog: python-v1.0.0...python-v1.0.1

python-v1.0.0: Zero to One

It only took us 5 years, but we made it! You can find the upgrade guide here.

Performance improvements

• refactor: async writer + multi-part by @ion-elgreco in #3255
• perf: use lazy sync reader by @ion-elgreco in #3338

New features

• feat: remove optimize operations when building without Apache Datafusion by @rtyler in #3290
• feat(api): add rustls and native-tls features by @zeevm in #3335
• feat!: update storage configuration system by @roeap in #3383
• feat: derive macro for config implementations by @roeap in #3389
• feat: upgrade to DataFusion 47.0.0 by @alamb in #3378
• feat: introduce VacuumMode::Full for cleaning up orphaned files by @rtyler in #3368
• feat: during LakeFS file operations, skip merge when 0 changes by @smeyerre in #3346
• feat: added a check for gc code to run by @JustinRush80 in #3419
• feat: spawn io with spawn service by @ion-elgreco in #3426
• feat: optimize datafusion predicate pushdown and partition pruning by @rtyler in #3436
• feat: expose kernel Engine on LogStore by @roeap in #3446
• refactor: remove pyarrow dependency by @ion-elgreco in #3459
• feat: write checkpoints with kernel by @roeap in #3466
• feat: add a table description and name to the Delta Table from Python by @fvaleye in #3464
• refactor!: remove and deprecate some python methods by @roeap in #3488

Bug Fixes

• fix: use field physical name when resolving partition columns by @zeevm in #3349
• fix(pandas): retain pyarrow decimal datatype in to_pandas() by adding types_mapper to prevent precision loss by @Abhishek1005 in #3296
• fix: prevent panics when peek_next_commit() encounters invalid data by @rtyler in #3308
• fix: serialize empty deletionVector in add actions as absent by @rtyler in #3309
• fix: stats column binary_column has unsupported type binary by @omkar-foss in #3146
• fix: check for all known valid delta files in is_deltatable by @umartin in #3318
• fix: block_in_place to allow nested tasks by @ion-elgreco in #3324
• fix: parse snapshot by @ion-elgreco in #3355
• fix: added restored metadata as action to the next committed version by @Nordalf in #3303
• fix: parse unconventional logs by @roeap in #3373
• fix: clippy warnings by @alamb in #3390
• fix: the default target size should be 100MB by @HiromuHota in #3404
• fix: if field contains space in constraint expression, checks will fail by @Nordalf in #3374
• fix: build Unity Catalog crate without DataFusion by @linhr in #3420
• fix: drop column update by @ion-elgreco in #3416
• fix: ignore temp log entries by @corwinjoy in #3423
• fix: use more accurate log path parsing by @roeap in #3461
• fix: correct spelling errors found by CI spell checker by @fvaleye in #3465
• fix: schema conversion, add conversion test cases by @ion-elgreco in #3468
• fix: set casting safe param to False by @ion-elgreco in #3481
• fix: ensure projecting only columns that exist in new files afte sche… by @alexwilcoxson-rel in #3487

Other Changes

• refactor: drop pyarrow support, restructure python modules by @ion-elgreco in #3285
• chore: bump python version for release by @rtyler in #3291
• chore: use flags for apple arm64 by @ion-elgreco in #3213
• chore: upgrade the kernel version and bump our majorish versions too by @rtyler in #3289
• chore: upgrade to DataFusion 46.0.0 by @alamb in #3261
• refactor: add 'cloud' feature to 'core' to enable 'cloud' on 'object_store' only when needed by @zeevm in #3332
• docs: update dataFusion integration example by @riziles in #3343
• refactor(python): improve typing, linting by @ion-elgreco in #3344
• chore: remove pyarrow upper by @ion-elgreco in #3325
• chore: improve io error msg by @ion-elgreco in #3328
• docs: update merge-tables.md with "Optimizing Merge Performance" section by @ldacey in #3351
• docs: add example how to authenticate using Azure CLI for Azure ADSL integration by @DanielBertocci in #3357
• chore: remove cdf feature by @ion-elgreco in #3365
• fix: correct Python docs for incremental compaction on OPTIMIZE by @roykim98 in #3301
• chore: fix some minor build warnings by @rtyler in #3366
• refactor: move transaction module to kernel by @roeap in #3380
• chore: clippy by @roeap in #3379
• chore: move proofs into dedicated folder by @roeap in #3381
• refactor!: move storage module into logstore by @roeap in #3382
• chore: put a couple symbols behind the right feature gate by @rtyler in #3393
• chore: update delta_kernel to 0.10.0 by @zachschuermann in #3403
• refactor: make "cloud" feature in object_store optional by @zeevm in #3398
• chore: bump versions of rust crates for another release party by @rtyler in #3406
• chore: commit the contents of the 0.26.0 release by @rtyler in #3408
• chore: reduce scope of feature flags and compilation requirements for subcrates by @rtyler in #3409
• chore(deps): update sqlparser requirement from 0.53.0 to 0.56.0 by @dependabot in #3413
• chore(deps): update foyer requirement from 0.16.1 to 0.17.0 by @dependabot in #3412
• chore: bringing dat integration testing in ahead of kernel replay by @rtyler in #3411
• chore: missed a version bump for core by @rtyler in #3415
• chore: include license file in deltalake-derive crate by @ankane in #3417
• chore(deps): bump foyer to v0.17.2 to prevent from wrong result by @MrCroxx in #3428
• chore: bump crate versions which are due for release by @rtyler in #3430
• chore: rely on the testing during coverage generation to speed up tests by @rtyler in #3431
• chore: make codecov more vigorously enforced to help ensure quality by @rtyler in #3434
• chore: prepare py-1.0 release by @ion-elgreco in #3435
• chore: experiment with using sccache in GitHub Actions by @rtyler in #3437
• chore: remove unused code and deps by @roeap in #3441
• chore: minor table module refactors by @rtyler in #3442
• docs: add 1.0.0 migration guide by @ion-elgreco in #3443
• refactor: more specific factory parameter names by @roeap in #3445
• refactor: use LogStore in Snapshot / LogSegment APIs by @roeap in #3452
• test: avoid circular dependency with core/test crates by @roeap in #3450
• chore: ensuring default builds work without datafusion by @rtyler in #3453
• ci: add spellchecker to pr tests by @roeap in #3457
• chore: mark more tests which require datafusion by @rtyler in #3458
• refactor: use full paths in log processing by @roeap in #3456
• chore: set correct markers by @ion-elgreco in https://github.com/delta-io/...

python-v0.25.5

What's Changed

• chore: relax the pyarrow constraints now that 19.0.1 is out by @rtyler in #3361

Full Changelog: python-v0.25.4...python-v0.25.5

python-v0.25.4

What's Changed

• chore: more manual inline format args by @nyurik in #3251
• fix: chrono 0.4.40 causes disambiguation syntax when building arrow deps by @Nordalf in #3272
• fix(rust): on write, have a schema evolution maintain metadata table id by @liamphmurphy in #3275
• chore: remove unused dev dependency on home by @alamb in #3277
• refactor: replaced asterisk with constraint name in get_constraints by @Nordalf in #3270
• refactor: simplify expressions by @ion-elgreco in #3281
• fix: compare before after schema before creating action by @ion-elgreco in #3282
• fix: timestamp truncation in stats parsed by @ion-elgreco in #3288

Full Changelog: python-v0.25.2...python-v0.25.4

python-v0.25.2

New features

• feat: set clientVersion for python binding by @ion-elgreco in #3242

Bug Fixes

• fix: aarch64 ssl handshakes by @ion-elgreco in #3263
• fix: enable missing cloud features for uc crate by @omkar-foss in #3260
• fix: refresh snapshot after vacuuming logs by @ion-elgreco in #3252
• fix: use tableprovider scan in z-order by @ion-elgreco in #3238
• fix: commitlint config by @roeap in #3268

Other Changes

• chore: inline format args to improve readability by @nyurik in #3154
• chore: update readme for Generated columns by @ion-elgreco in #3247
• docs: update CONTRIBUTING.md and Makefile to account for switch to uv by @adamreeve in #3257
• chore: fix some compiler warnings by @alamb in #3262
• chore: fix clippy warnings on main by @alamb in #3264
• chore: use builder API to create FileScanConfig by @alamb in #3266
• chore: update tests to use Column::new and other expr_fn functions by @alamb in #3265

New Contributors

• @adamreeve made their first contribution in #3257

Full Changelog: python-v0.25.1...python-v0.25.2

python-v0.25.0: Improved memory writer, schema evolution `merge` and UC support

Highlights

• Streamed execution in write and merge (lower memory pressure)
• Overall improved memory allocation
• Schema evolution in MERGE
• Unity catalog support through uc:// uri-prefix

Performance improvements

• feat: streamed execution in MERGE by @ion-elgreco in #3145
• feat: streamed execution, writer refactor, simplified generated columns and schema evolution by @ion-elgreco in #3229
• refactor: logical plans in writer by @ion-elgreco in #3141
• chore(python): choose performance over compile time in release mode by @ion-elgreco in #3177
• chore(python): use custom global alloc by @ion-elgreco in #3176

New features

• feat: add schema evolution to merge statement by @JustinRush80 in #3136
• feat(python): add capability to read unity catalog (uc://) uris by @omkar-foss in #3113
• feat: support reading tables via Unity Catalog provided credentials by @hntd187 in #3078
• feat: added workspace oauth source for UC by @hntd187 in #3152
• feat: update metadata field builder by @Nordalf in #3149
• feat: set column metadata from python by @umartin in #3189
• feat(disable conflict checker): start counting concurrent retries at 0 and disable conflict checker with maxRetries=0 by @daphnenhuch-at in #3204
• feat: make Add:get_stats public by @jkylling in #3216
• feat: configurable column encoding for parquet checkpoint files by @dmunch in #3214
• feat: return metrics about the commit by @HawaiianSpork in #3202
• feat: cdf tableprovider with predicate pushdown support by @ion-elgreco in #3220

Bug Fixes

• fix: properly encode metadata field values when creating actions by @rtyler in #3183
• fix: only use stats for required cols by @ion-elgreco in #3210
• fix: load cdf latest version by @ion-elgreco in #3218
• fix: adhere write stats configuration by @ion-elgreco in #3209
• fix(rust, python): treat FSCK files_removed as strings by @liamphmurphy in #3219
• fix: update snapshot after conflict check by @ion-elgreco in #3205

Other Changes

• chore: preparing the 0.24 release of the Rust crates` by @rtyler in #3132
• chore: fix typos by @roeap in #3138
• refactor: combine normal and cdf plan until write for merge by @ion-elgreco in #3142
• chore: remove lazy_static and once_cell dependencies by @nyurik in #3148
• docs: Fix broken markdown syntax by @hattajr in #3134
• chore: minor changes to allow 0.24.0 to be published by @rtyler in #3158
• chore: prevent the pr actions from filing on merge_group by @rtyler in #3161
• docs: fix broken links by @ion-elgreco in #3179
• chore: update to DataFusion 45.0.0 / arrow/parquet 54.0.0 by @alamb in #3175
• chore: adopt the released datafusion 45 release by @rtyler in #3195
• fix: removing openssl from the dependency tree by @rtyler in #3160
• chore: remove the unnecessary hashbrown dependency by @rtyler in #3198
• chore: switch to cache@v4 by @rtyler in #3230
• fix(typo): s/implemtnation/implementation/ by @akesling in #3227
• chore: bump python 0.25 by @ion-elgreco in #3233
• chore(python): set streaming opt-in by @ion-elgreco in #3178

New Contributors

• @nyurik made their first contribution in #3148
• @hattajr made their first contribution in #3134
• @JustinRush80 made their first contribution in #3136
• @umartin made their first contribution in #3189
• @daphnenhuch-at made their first contribution in #3204
• @dmunch made their first contribution in #3214
• @akesling made their first contribution in #3227

Full Changelog: python-v0.24.0...python-v0.25.0

python-v0.24.0: LakeFS integration + GeneratedColumns

New features

• feat: lakefs integration by @ion-elgreco in #3103
• feat: generated columns by @ion-elgreco in #3123
• feat: retryable config enablement through storage_options by @ion-elgreco in #3129
• feat(python, rust): add statistics_enabled to ColumnProperties by @maxitg in #3126

Bug Fixes

• fix: aws credential resolving, reduce api calls by @ion-elgreco in #3107
• fix: reading cdf from a checkpointed table by @hntd187 in #3110
• fix: ignore crc files when checking if provided path correspond to a valid delta table by @guillotjulien in #3122
• fix: fmt scalar view values by @ion-elgreco in #3130
• fix: docs rendering by @Abdullahsab3 in #3120

Other Changes

• refactor: use kernel table features by @ion-elgreco in #3117
• chore: update readme with latest state of project by @ion-elgreco in #3118
• chore(deps): update itertools requirement from 0.13 to 0.14 by @dependabot in #3104
• chore: allow kernel::visitors to be public for Snapshot usage by @rtyler in #2966
• feat(docs): minor documentation typo by @Milias in #3131
• chore: 0.24 py release by @ion-elgreco in #3127
• chore: raise patch versions for rust crate release by @rtyler in #3114

New Contributors

• @guillotjulien made their first contribution in #3122
• @Milias made their first contribution in #3131

Full Changelog: python-v0.23.2...python-v0.24.0

python-v0.23.2

What's Changed

• fix: introduce a reproduction case for List casting with polars by @rtyler in #3093
• feat(python): add except_cols argument to when_matched_update_all and when_not_matched_insert_all for excluding specific columns by @mrjsj in #3098
• feat(python): add sql() and show() convenience method to QueryBuilder by @DevChrisCross in #3100
• docs: rust usage documentation by @Abdullahsab3 in #3089
• feat: allow multiple Python threads to work with a single DeltaTable instance by @rtyler in #3101

New Contributors

• @DevChrisCross made their first contribution in #3100
• @Abdullahsab3 made their first contribution in #3089

Full Changelog: python-v0.23.1...python-v0.23.2

python-v0.23.1

What's Changed

• fix: scalar extraction view types by @ion-elgreco in #3095

Full Changelog: python-v0.23.0...python-v0.23.1

python-v0.23.0: conditional put for S3 by default

New features

• feat: add out_of_range flag to load_cdf by @pblocz in #3040
• refactor: make put-if-absent default for S3 stores by @ion-elgreco in #3091

Bug Fixes

• fix: add explicit type casts while merge by @t1g0rz in #3050
• fix: add decimal expr parsing by @ion-elgreco in #3090
• fix: raise during checkpoint creating with without_files by @ion-elgreco in #3059
• fix: use of deprecated DeltaTable::peek_next_commit by @jkylling in #3080
• fix: remove println from regular_rename func by @tom1919 in #3071
• fix: use projected_table_schema for projection in DeltaSchemaAdapter by @jkylling in #3068
• fix: use correct aws_endpoint key by @ion-elgreco in #3072
• fix: add test for Parquet column pruning in scans by @jkylling in #3079
• fix: allow more than 15 concurrent transactions to have been committed by @rtyler in #3067

Other Changes

• chore: bump delta-kernel to 0.5.0 by @roeap in #3051
• chore: bump kernel 0.6 and datafusion 44 by @ion-elgreco in #3087
• chore: use uv by @thomas-chauvet in #3042
• Add steps to install uv in venv by @FranArenas in #3085
• feat: move unity catalog integration to it's own crate by @hntd187 in #3044
• chore: bump versions for next release(s) by @rtyler in #3061
• chore: create a new minor version with the MSRV upgrade by @rtyler in #3069

New Contributors

• @pblocz made their first contribution in #3040
• @tom1919 made their first contribution in #3071
• @FranArenas made their first contribution in #3085

Full Changelog: python-v0.22.3...python-v0.23.0