feat(PECOBLR-2328): add _retry_server_directed_only connection parameter#1363
Open
tejassp-db wants to merge 4 commits intomainfrom
Open
feat(PECOBLR-2328): add _retry_server_directed_only connection parameter#1363tejassp-db wants to merge 4 commits intomainfrom
_retry_server_directed_only connection parameter#1363tejassp-db wants to merge 4 commits intomainfrom
Conversation
Bump databricks-sql-connector to >=4.1.5 and expose the new _retry_server_directed_only parameter (default: false) to prevent duplicate writes caused by blind 503 retries on ExecuteStatement. Users can opt in via connection_parameters in profiles.yml. PECOBLR-2328 Co-authored-by: Isaac
Co-authored-by: Isaac
tejassp-db
changed the title
tejassp-db
commented
Mar 24, 2026
| "click>=8.2.0, <9.0.0", | ||
| "databricks-sdk>=0.68.0, <0.78.0", | ||
| "databricks-sql-connector[pyarrow]>=4.1.1, <4.1.4", | ||
| "databricks-sql-connector[pyarrow]>=4.1.5, <4.2.0", |
Collaborator
Author
There was a problem hiding this comment.
@jprakash-db Is this acceptable - since we don't want it to move beyond 4.2.0?
Collaborator
There was a problem hiding this comment.
- I think we should have tighter pinning today <4.1.6
- Any reason to update the lower version as well to 4.1.5 ? We can do this here but not required and we never know if this might break some env. IMO let us not update the minimum supported version
Coverage report
Click to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||
sd-db
changed the title
_retry_server_directed_only connection parameter
sd-db
reviewed
Apr 15, 2026
Collaborator
sd-db
left a comment
sd-db
left a comment
There was a problem hiding this comment.
Let us also update the CHANGELOG.md to list that we are updating the python-sql-connector dependency
| "click>=8.2.0, <9.0.0", | ||
| "databricks-sdk>=0.68.0, <0.78.0", | ||
| "databricks-sql-connector[pyarrow]>=4.1.1, <4.1.4", | ||
| "databricks-sql-connector[pyarrow]>=4.1.5, <4.2.0", |
Collaborator
There was a problem hiding this comment.
- I think we should have tighter pinning today <4.1.6
- Any reason to update the lower version as well to 4.1.5 ? We can do this here but not required and we never know if this might break some env. IMO let us not update the minimum supported version
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
databricks-sql-connectordependency to>=4.1.5, <4.2.0to pick up the new_retry_server_directed_onlyparameter (databricks-sql-python#756)_retry_server_directed_onlyas a default connection parameter (default:false, opt-in) incredentials.py__pre_deserialize__defaults and adapter-level pass-through todbsql.connect()Context
Customers using
use_materialization_v2: trueexperience duplicate rows when the server returns HTTP 503 after already committing an INSERT. The connector blindly retries, causing data to be written twice. With_retry_server_directed_only: true, retries only occur when the server explicitly sends aRetry-Afterheader, preventing duplicate writes from infrastructure-level 503s.Users can opt in via
profiles.yml:Test plan
TestDatabricksCredentialsPreDeserialize— 5 tests for__pre_deserialize__default/override behaviortest_databricks_sql_connector_default_retry_params— verifies defaults reachdbsql.connect()test_databricks_sql_connector_retry_server_directed_only_opt_in— verifies user opt-in flows throughhatch run unit)