use dependency groups to seperate local spark and databricks-connect (#23)

* use dependency groups to seperate local spark and databricks-connect

* remove dbt from local spark

* update lock file

* properly handle dev dependencies

* ensure no-dev for local spark

Author: frizzleqq

.github/workflows/ci.yml

@@ -32,13 +32,13 @@ jobs:
         java-version: 17
         distribution: "zulu"
     - name: Install the project
-      run: uv sync --locked --all-extras
+      run: uv sync --locked --group dev-spark --no-dev
     - name: Run code checks
-      run: uv run ruff check
+      run: uv run --no-dev ruff check
     - name: Check code formatting
-      run: uv run ruff format --check
+      run: uv run --no-dev ruff format --check
     - name: Run tests
-      run: uv run pytest -v
+      run: uv run --no-dev pytest -v
 
   ci-databricks:
     needs: ci-local
@@ -64,12 +64,8 @@ jobs:
       with:
         version: 0.280.0
     - name: Install the project
-      run: uv sync --locked --all-extras
-    - name: Install Databricks Connect
-      run: |
-        uv pip uninstall pyspark
-        uv pip install databricks-connect==17.2.*
+      run: uv sync --locked
     - name: Check Databricks CLI
       run: databricks current-user me
     - name: Run tests
-      run: uv run --no-sync pytest -v
+      run: uv run pytest -v

.github/workflows/deploy-test.yml

@@ -34,7 +34,7 @@ jobs:
       with:
         version: 0.280.0
     - name: Install the project
-      run: uv sync --locked --all-extras
+      run: uv sync --locked
     - name: Check Databricks CLI
       run: databricks current-user me
     - name: Log Bundle Changes

.github/workflows/release.yml

@@ -35,7 +35,7 @@ jobs:
       with:
         version: 0.280.0
     - name: Install the project
-      run: uv sync --locked --all-extras
+      run: uv sync --locked
     - name: Check Databricks CLI
       run: databricks current-user me
     - name: Log Bundle Changes

.github/workflows/validate-bundle.yml

@@ -33,7 +33,7 @@ jobs:
       with:
         version: 0.280.0
     - name: Install the project
-      run: uv sync --locked --all-extras
+      run: uv sync --locked
     - name: Check Databricks CLI
       run: databricks current-user me
     - name: Validate Databricks Bundle

AGENTS.md

@@ -10,16 +10,8 @@ This repo uses Databricks CLI to deploy a Databricks Asset Bundle.
 - `tests`: Unit tests for the Python project
 
  ## Setup commands
-- Install deps: `uv sync --locked --all-extras`
-- Run code checks: `uv run ruff check`
-- Check code formatting: `uv run ruff format --check`
-- Run tests: `uv run pytest -v`
- 
-## Unit-Tests
-- On Linux systems, run: `uv run pytest -v`
-- On Windows systems, run:
-    ```powershell
-    uv pip uninstall pyspark
-    uv pip install databricks-connect==17.2.*
-    uv run --no-sync pytest -v
-    ```
+- Install deps: `uv sync --locked --group dev-spark --no-dev`
+- Run code checks: `uv run --no-dev ruff check`
+- Check code formatting: `uv run --no-dev ruff format --check`
+- Run tests: `uv run --no-dev pytest -v`
+ 

README.md

@@ -49,12 +49,12 @@ A script exists to set up the Workspace (Free Edition) as described in the [Setu
 
 ### Setup environment
 
-Sync entire `uv` environment with all optional dependency groups:
+Sync `uv` environment with `dev` (includes databricks-connect) dependencies:
 ```bash
-uv sync --all-extras
+uv sync --locked
 ```
 
-> **Note:** we install Databricks Connect in a follow-up step
+> **Note:** For local Spark use `uv sync --group dev-spark --no-dev` instead.
 
 #### (Optional) Activate virtual environment
 
@@ -70,35 +70,19 @@ Windows:
 
 ### Databricks Connect
 
-Install `databricks-connect` in active environment. This requires authentication being set up via Databricks CLI.
-
-```bash
-uv pip uninstall pyspark
-uv pip install databricks-connect==17.2.*
-```
-
-**Option 2: Run with temporary dependency**
-```bash
-uv run --with databricks-connect==17.2.* pytest
-```
-
-> **Note:** For Databricks Runtime Serverless v4
-
+The `dev` dependency group includes `databricks-connect` for remote Spark execution. This requires authentication being set up via Databricks CLI.
 
 See https://docs.databricks.com/aws/en/dev-tools/vscode-ext/ for using Databricks Connect extension in VS Code.
 
 ### Unit-Tests
 
 ```bash
-# in case databricks-connect is installed, --no-sync prevents reinstalling pyspark
-uv run --no-sync pytest -v
+uv run pytest -v
 ```
 
-Based on whether Databricks Connect is enabled or not the Unit-Tests use a Databricks Cluster or start a local Spark session with Delta support.
+Based on whether Databricks Connect or local Spark is installed, the Unit-Tests use a Databricks Cluster or start a local Spark session with Delta support.
 * On Databricks the unit-tests currently assume the catalog `lake_dev` exists.
 
-> **Note:** For local Spark Java is required. On Windows Spark/Delta requires HADOOP libraries and generally does not run well, opt for `wsl` instead.
-
 ### Checks
 
 ```bash

pyproject.toml

@@ -19,19 +19,31 @@ dependencies = [
     "databricks-sdk>=0.41, <0.68.0", # pinning from dbt-databricks
 ]
 
-[project.optional-dependencies]
+[dependency-groups]
+# Development with databricks-connect
+dev = [
+    { include-group = "databricks" },
+    { include-group = "dbt" },
+    { include-group = "tooling" },
+    "databricks-connect>=17.0,<17.4",
+]
+# Development with local Spark
+dev-spark = [
+    { include-group = "databricks" },
+    { include-group = "tooling" },
+    "delta-spark==4.0.*",
+    "pyspark==4.0.*",
+]
 # Databricks runtime dependencies (preinstalled on cluster)
 databricks = [
-    "delta-spark==4.0.*",
     "pydantic==2.10.6",
-    "pyspark==4.0.*",
 ]
 # dbt dependencies
 dbt = [
     "dbt-databricks==1.11.*",
 ]
-# Development & Testing
-dev = [
+# Development tooling (shared across dev environments)
+tooling = [
     "databricks-bundles==0.280.*", # For Python-based Workflows
     "mypy", # Type hints
     "pip", # Databricks extension needs it