dlt version

1.17.1

Describe the problem

When attempting to run dlt on a job cluster on Databricks, using runtime 16.4 LTS , I encountered the following error:

Unexpected internal error when monkey patching dlt module: cannot import name 'overrides' from partially initialized module 'dlt' (most likely due to a circular import) (/databricks/python/lib/python3.12/site-packages/dlt/__init__.py)

Expected behavior

dlt runs on as it did on previous Databricks runtimes

Steps to reproduce

Use the init script from the current documentation on a Databricks job cluster with runtime version 16.4 LTS.

Operating system

Linux, Other

Runtime environment

Other

Python version

3.12

dlt data source

No response

dlt destination

No response

Other deployment details

No response

Additional information

I had already altered one of the paths in the init script to get it to work on 15.4 LTS in the past. This was based on a discussion I found in some reddit post. So I'm assuming the paths have changed again.