Skip to content
This repository was archived by the owner on Sep 4, 2024. It is now read-only.

Commit 3c858a4

Browse files
pankajkotipankajkoti
authored
Deprecate the provider and proxy to upstream Airflow provider (#84)
As part of Astronomer's internal plans and decisions, we've decided to contribute the existing functionality provided by the operators and plugins in this repository to the official Apache Airflow Databricks provider. To achieve this, we submitted the following PRs to the Airflow provider: 1. apache/airflow#39178 2. apache/airflow#39771 3. apache/airflow#40013 4. apache/airflow#40724 5. apache/airflow#39295 All functionality has now been contributed to the Airflow Databricks provider, and ongoing support will be maintained there. As a result, we're deprecating the operators and plugins in this repository. Users are encouraged to transition to the official Apache Airflow Databricks provider as soon as possible. The migration process is straightforward—simply update the import path to point to the Airflow provider and ensure that you install `apache-airflow-providers-databricks>=6.8.0`, which includes all the contributions mentioned above. closes: astronomer/issues-airflow#715
1 parent d49e88c commit 3c858a4

File tree

15 files changed

+192
-3249
lines changed

15 files changed

+192
-3249
lines changed

‎.github/workflows/ci.yml

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -78,7 +78,7 @@ jobs:
7878
fail-fast: false
7979
matrix:
8080
python: ['3.8', '3.9', '3.10']
81-
airflow: [2.5]
81+
airflow: [2.7]
8282
if: >-
8383
github.event_name == 'push' ||
8484
(
@@ -125,7 +125,7 @@ jobs:
125125
fail-fast: false
126126
matrix:
127127
python: ['3.8', '3.9', '3.10']
128-
airflow: ['2.3', '2.4', '2.5', '2.6', '2.7', '2.8']
128+
airflow: ['2.7', '2.8']
129129

130130
if: >-
131131
github.event_name == 'push' ||
@@ -172,7 +172,7 @@ jobs:
172172
path: ./.coverage
173173
env:
174174
DATABRICKS_CONN_TOKEN: ${{ secrets.DATABRICKS_CONN_TOKEN }}
175-
DATABRICKS_CONN_HOST: {{ secrets.DATABRICKS_CONN_HOST }}
175+
DATABRICKS_CONN_HOST: ${{ secrets.DATABRICKS_CONN_HOST }}
176176
DATABRICKS_CONN: ${{ secrets.AIRFLOW_CONN_DATABRICKS_DEFAULT }}
177177

178178

‎CHANGELOG.rst

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,14 @@
11
Changelog
22
=========
33

4+
0.3.0rc1 (06-08-24)
5+
-------------------
6+
7+
Deprecations
8+
9+
* Deprecate the provider and proxy instantiations to upstream official Apache Airflow Databricks provider (PR `#84 <https://github.com/astronomer/astro-provider-databricks/pull/84>`_ by @pankajkoti)
10+
11+
412
0.2.2 (16-04-24)
513
----------------
614

‎README.md

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,19 @@
1+
## Deprecation Notice
2+
3+
With the release ``0.3.0`` of the ``astro-provider-databricks`` package, this provider stands deprecated and will
4+
no longer receive updates. We recommend migrating to the official ``apache-airflow-providers-databricks>= 6.8.0`` for the latest features and support.
5+
For the operators and sensors that are deprecated in this repository, migrating to the official Apache Airflow Databricks Provider
6+
is as simple as changing the import path in your DAG code as per the below examples.
7+
8+
| Previous import path used (Deprecated now) | Suggested import path to use |
9+
|-------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------|
10+
| `from astro_databricks.operators.notebook import DatabricksNotebookOperator` | `from airflow.providers.databricks.operators.databricks import DatabricksNotebookOperator` |
11+
| `from astro_databricks.operators.workflow import DatabricksWorkflowTaskGroup` | `from airflow.providers.databricks.operators.databricks_workflow import DatabricksWorkflowTaskGroup` |
12+
| `from astro_databricks.operators.common import DatabricksTaskOperator` | `from airflow.providers.databricks.operators.databricks import DatabricksTaskOperator` |
13+
| `from astro_databricks.plugins.plugin import AstroDatabricksPlugin` | `from airflow.providers.airflow.providers.databricks.plugins.databricks_workflow import DatabricksWorkflowPlugin` |
14+
15+
# Archives
16+
117
<h1 align="center">
218
Databricks Workflows in Airflow
319
</h1>

‎docs/index.rst

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,30 @@
1+
.. warning::
2+
All the operators and their functionality within this repository have been deprecated and will not receive further updates.
3+
Read more about the deprecation in the `Deprecation Notice` section below.
4+
5+
Deprecation Notice
6+
------------------
7+
8+
With the release ``0.3.0`` of the ``astro-provider-databricks`` package, this provider stands deprecated and will
9+
no longer receive updates. We recommend migrating to the official ``apache-airflow-providers-databricks>=6.8.0`` for the latest features and support.
10+
For the operators and sensors that are deprecated in this repository, migrating to the official Apache Airflow Databricks Provider
11+
is as simple as changing the import path in your DAG code as per the below examples.
12+
13+
.. list-table:: Import paths to change for migrating to the official Apache Airflow Databricks Provider
14+
:header-rows: 1
15+
16+
* - Previous import path used
17+
- Newer import path to use
18+
* - from astro_databricks.operators.notebook import DatabricksNotebookOperator
19+
- from airflow.providers.databricks.operators.databricks import DatabricksNotebookOperator
20+
* - from astro_databricks.operators.workflow import DatabricksWorkflowTaskGroup
21+
- from airflow.providers.databricks.operators.databricks_workflow import DatabricksWorkflowTaskGroup
22+
* - from astro_databricks.operators.common import DatabricksTaskOperator
23+
- from airflow.providers.databricks.operators.databricks import DatabricksTaskOperator
24+
* - from astro_databricks.plugins.plugin import AstroDatabricksPlugin
25+
- from airflow.providers.airflow.providers.databricks.plugins.databricks_workflow import DatabricksWorkflowPlugin
26+
27+
128
Astro Databricks Provider
229
=========================
330

‎pyproject.toml

Lines changed: 2 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -35,12 +35,8 @@ classifiers = [
3535
"Programming Language :: Python :: 3.10",
3636
]
3737
dependencies = [
38-
"apache-airflow>=2.3",
39-
"databricks-sql-connector>=2.0.4;python_version>='3.10'",
40-
"databricks-cli>=0.17.7",
41-
"apache-airflow-providers-databricks>=2.2.0",
42-
"mergedeep",
43-
"pydantic>=1.10.0",
38+
"apache-airflow>=2.7",
39+
"apache-airflow-providers-databricks>=6.8.0",
4440
]
4541

4642
[project.optional-dependencies]

‎src/astro_databricks/__init__.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
from astro_databricks.operators.notebook import DatabricksNotebookOperator
44
from astro_databricks.operators.workflow import DatabricksWorkflowTaskGroup
55

6-
__version__ = "0.2.2"
6+
__version__ = "0.3.0rc1"
77
__all__ = [
88
"DatabricksNotebookOperator",
99
"DatabricksWorkflowTaskGroup",

0 commit comments

Comments
 (0)