feat: add classic cluster option for Switch job deployment by hiroyukinakazato-db · Pull Request #2222 · databrickslabs/lakebridge

hiroyukinakazato-db · 2026-01-09T14:37:35Z

Changes

What does this PR do?

Add support for classic job cluster via environment variable LAKEBRIDGE_CLUSTER_TYPE=CLASSIC for Switch jobs. This provides a workaround for workspaces where serverless is not available.

Relevant implementation details

Added switch_use_serverless flag to LakebridgeConfiguration (not persisted, used only during install)
Environment variable LAKEBRIDGE_CLUSTER_TYPE=CLASSIC triggers classic cluster mode
SwitchDeployment.install() now accepts use_serverless parameter
When use_serverless=False, creates a job cluster with:
- LTS Spark version
- Single worker node (16GB+ memory)
- USER_ISOLATION data security mode

Usage

# Default: Serverless (no change needed)
databricks labs lakebridge install-transpile --include-llm-transpiler true

# Classic cluster
LAKEBRIDGE_CLUSTER_TYPE=CLASSIC databricks labs lakebridge install-transpile --include-llm-transpiler true

Caveats/things to watch out for when reviewing:

Default behavior remains serverless (backwards compatible)
Classic cluster sizing (16GB+ memory, 1 worker) is fixed for now

Linked issues

Resolves #2219

Functionality

added relevant user documentation
added new CLI command
modified existing command: databricks labs lakebridge install-transpile

Tests

manually tested
added unit tests
added integration tests

Add prompt during install-transpile to allow users to choose between serverless compute (default) and classic job cluster for Switch jobs. This provides a workaround for workspaces where serverless access controls via usage policies require explicit policy selection.

github-actions · 2026-01-09T14:42:10Z

✅ 129/129 passed, 6 flaky, 5 skipped, 16m47s total

Flaky tests:

🤪 test_installs_and_runs_local_bladebridge (20.245s)
🤪 test_installs_and_runs_pypi_bladebridge (22.814s)
🤪 test_transpiles_informatica_to_sparksql_non_interactive[True] (18.143s)
🤪 test_transpile_teradata_sql (20.382s)
🤪 test_transpiles_informatica_to_sparksql_non_interactive[False] (3.921s)
🤪 test_transpile_teradata_sql_non_interactive[True] (5.615s)

_{Running from acceptance #3429}

codecov · 2026-01-09T14:42:20Z

Codecov Report

❌ Patch coverage is 94.73684% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 64.07%. Comparing base (254b5d3) to head (f653402).
⚠️ Report is 6 commits behind head on main.

Files with missing lines	Patch %	Lines
src/databricks/labs/lakebridge/cli.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2222      +/-   ##
==========================================
+ Coverage   64.05%   64.07%   +0.02%     
==========================================
  Files         100      100              
  Lines        8624     8633       +9     
  Branches      893      894       +1     
==========================================
+ Hits         5524     5532       +8     
- Misses       2928     2929       +1     
  Partials      172      172

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…Switch

hiroyukinakazato-db · 2026-01-13T02:13:09Z

Manual Test Evidence

Test 1: Install with serverless compute (default)

$ databricks labs lakebridge install-transpile --include-llm-transpiler true --profile e2-demo-tokyo
Use serverless compute for Switch job? (no = use classic job compute instead) (default: yes): yes
09:51:39     INFO [d.labs.lakebridge] Skipping transpile configuration prompts for LLM transpiler installation.
09:51:40     INFO [d.l.l.transpiler.installers] databricks-bb-plugin v0.1.24 already installed
09:51:40     INFO [d.l.l.transpiler.installers] Databricks databricks-morph-plugin transpiler v0.6.16 already installed
09:51:40     INFO [d.l.lakebridge.install] Configuring lakebridge `transpile`.
09:51:41     INFO [d.l.lakebridge.install] Lakebridge `transpile` is already installed on this workspace.
09:52:19     INFO [d.l.lakebridge.install] Installing Switch transpiler to workspace.
09:52:19     INFO [d.l.l.deployment.switch] Copying resources to workspace: /Users/hiroyuki.nakazato@databricks.com/.lakebridge/switch
09:52:35     INFO [d.l.l.deployment.switch] Switch job created/updated: https://e2-demo-tokyo.cloud.databricks.com/jobs/569534437124164
09:52:35     INFO [d.l.lakebridge.install] Installation completed successfully! Please refer to the documentation for the next steps.

Installation result

Test 2: Install with classic job compute

$ databricks labs lakebridge install-transpile --include-llm-transpiler true --profile e2-demo-tokyo
Use serverless compute for Switch job? (no = use classic job compute instead) (default: yes): no
09:54:02     INFO [d.labs.lakebridge] Skipping transpile configuration prompts for LLM transpiler installation.
09:54:02     INFO [d.l.l.transpiler.installers] databricks-bb-plugin v0.1.24 already installed
09:54:03     INFO [d.l.l.transpiler.installers] Databricks databricks-morph-plugin transpiler v0.6.16 already installed
09:54:03     INFO [d.l.lakebridge.install] Configuring lakebridge `transpile`.
09:54:03     INFO [d.l.lakebridge.install] Lakebridge `transpile` is already installed on this workspace.
09:54:39     INFO [d.l.lakebridge.install] Installing Switch transpiler to workspace.
09:54:39     INFO [d.l.l.deployment.switch] Copying resources to workspace: /Users/hiroyuki.nakazato@databricks.com/.lakebridge/switch
09:54:55     INFO [d.l.l.deployment.switch] Switch job created/updated: https://e2-demo-tokyo.cloud.databricks.com/jobs/569534437124164
09:54:55     INFO [d.l.lakebridge.install] Installation completed successfully! Please refer to the documentation for the next steps.

Installation result

Test 3: Job execution with classic job compute

$ databricks labs lakebridge llm-transpile \
  --input-source /Users/hiroyuki.nakazato/0/repos/lakebridge-switch/switch/examples/sql/mssql/input \
  --output-ws-folder /Workspace/Users/hiroyuki.nakazato@databricks.com/switch-io/mssql/output \
  --source-dialect mssql \
  --accept-terms true \
  --profile e2-demo-tokyo
Enter catalog name (default: lakebridge): hinak_catalog_aws_apne1
10:52:51     INFO [d.l.l.deployment.configurator] Found existing catalog `hinak_catalog_aws_apne1`
Enter schema name (default: switch):
10:53:01     INFO [d.l.l.deployment.configurator] Found existing schema `switch` in catalog `hinak_catalog_aws_apne1`
Enter volume name (default: switch_volume):
10:53:05     INFO [d.l.l.deployment.configurator] Found existing volume `switch_volume` in catalog `hinak_catalog_aws_apne1` and schema `switch`
Select a Foundation Model serving endpoint:
[0] [Recommended] databricks-claude-sonnet-4-5
...
Enter a number between 0 and 26: 0
10:53:17     INFO [d.l.l.transpiler.switch_runner] Uploading /Users/hiroyuki.nakazato/0/repos/lakebridge-switch/switch/examples/sql/mssql/input to /Volumes/hinak_catalog_aws_apne1/switch/switch_volume/input-20260113015317-klq3...
10:53:18     INFO [d.l.l.transpiler.switch_runner] Upload complete: /Volumes/hinak_catalog_aws_apne1/switch/switch_volume/input-20260113015317-klq3
10:53:18     INFO [d.l.l.transpiler.switch_runner] Triggering Switch job with job_id: 569534437124164
10:53:18     INFO [d.l.l.transpiler.switch_runner] Switch LLM transpilation job started: https://e2-demo-tokyo.cloud.databricks.com/jobs/569534437124164/runs/572505162987961

Execution result

gueniai · 2026-01-14T23:29:42Z

@hiroyukinakazato-db Can we change use-serverless to be a non-prompting param, like the include-llm-transpiler? By default it's set to true (unlike include...) and only if required, like in the case when customers have policies in place, they would turn it off?

sundarshankar89 · 2026-01-15T14:00:46Z

@hiroyukinakazato-db Can we change use-serverless to be a non-prompting param, like the include-llm-transpiler? By default it's set to true (unlike include...) and only if required, like in the case when customers have policies in place, they would turn it off?

I will recommend using environmental variables like LAKEBRIDGE_CLUSTER_TYPE=CLASSIC, we set up classic and this is non non-documented internal flag. One challenge using Classic is cluster sizing and type, that needs to be optimal.

gueniai · 2026-01-15T18:04:11Z

I like the idea of using an env variable. @hiroyukinakazato-db let us know what you think!

hiroyukinakazato-db · 2026-01-16T03:45:56Z

I like the idea of using an env variable. @hiroyukinakazato-db let us know what you think!
@sundarshankar89 @gueniai I also think it is good idea, I will try to update the code using this approach.

Replace interactive prompt with environment variable for Switch job compute type selection. Set LAKEBRIDGE_CLUSTER_TYPE=CLASSIC to use classic job cluster instead of serverless (default). This is an internal, non-documented flag for workspaces where serverless is not available.

hiroyukinakazato-db · 2026-01-16T05:34:30Z

Manual Test Evidence (Updated)

Updated to use environment variable approach and re-tested.

Branch and Commit

$ git branch --show-current && git log -1 --oneline
feat/switch-classic-cluster-support
afc781cf feat: use LAKEBRIDGE_CLUSTER_TYPE env var for classic cluster selection

Test 1: Install with serverless compute (default)

$ hatch run databricks labs lakebridge install-transpile --include-llm-transpiler true --profile e2-demo-tokyo
14:08:16     INFO [d.labs.lakebridge] Skipping transpile configuration prompts for LLM transpiler installation.
14:08:16     INFO [d.l.l.transpiler.installers] databricks-bb-plugin v0.1.24 already installed
14:08:16     INFO [d.l.l.transpiler.installers] Databricks databricks-morph-plugin transpiler v0.6.16 already installed
14:08:16     INFO [d.l.lakebridge.install] Configuring lakebridge `transpile`.
14:08:16     INFO [d.l.lakebridge.install] Lakebridge `transpile` is already installed on this workspace.
14:08:54     INFO [d.l.lakebridge.install] Installing Switch transpiler to workspace.
14:08:54     INFO [d.l.l.deployment.switch] Copying resources to workspace: /Users/hiroyuki.nakazato@databricks.com/.lakebridge/switch
14:09:11     INFO [d.l.l.deployment.switch] Switch job created/updated: https://e2-demo-tokyo.cloud.databricks.com/jobs/1002520435286464
14:09:11     INFO [d.l.lakebridge.install] Installation completed successfully! Please refer to the documentation for the next steps.

Installation result: Job created without job cluster (serverless)

Test 2: Install with classic job compute (using env var)

$ LAKEBRIDGE_CLUSTER_TYPE=CLASSIC hatch run databricks labs lakebridge install-transpile --include-llm-transpiler true --profile e2-demo-tokyo
14:09:36     INFO [d.labs.lakebridge] Skipping transpile configuration prompts for LLM transpiler installation.
14:09:36     INFO [d.l.l.transpiler.installers] databricks-bb-plugin v0.1.24 already installed
14:09:36     INFO [d.l.l.transpiler.installers] Databricks databricks-morph-plugin transpiler v0.6.16 already installed
14:09:36     INFO [d.l.lakebridge.install] Configuring lakebridge `transpile`.
14:09:36     INFO [d.l.lakebridge.install] Lakebridge `transpile` is already installed on this workspace.
14:10:14     INFO [d.l.lakebridge.install] Installing Switch transpiler to workspace.
14:10:14     INFO [d.l.l.deployment.switch] Copying resources to workspace: /Users/hiroyuki.nakazato@databricks.com/.lakebridge/switch
14:10:31     INFO [d.l.l.deployment.switch] Switch job created/updated: https://e2-demo-tokyo.cloud.databricks.com/jobs/1002520435286464
14:10:31     INFO [d.l.lakebridge.install] Installation completed successfully! Please refer to the documentation for the next steps.

Installation result: Job created with classic job cluster (LTS Spark, 16GB+ memory, USER_ISOLATION)

Test 3: Job execution with classic job compute

$ hatch run databricks labs lakebridge llm-transpile \
  --input-source /Users/hiroyuki.nakazato/0/repos/lakebridge-switch/switch/examples/sql/mssql/input \
  --output-ws-folder /Workspace/Users/hiroyuki.nakazato@databricks.com/switch-io/mssql/output \
  --source-dialect mssql \
  --accept-terms true \
  --profile e2-demo-tokyo
Enter catalog name (default: lakebridge): hinak_catalog_aws_apne1
14:11:32     INFO [d.l.l.deployment.configurator] Found existing catalog `hinak_catalog_aws_apne1`
Enter schema name (default: switch): switch
14:11:36     INFO [d.l.l.deployment.configurator] Found existing schema `switch` in catalog `hinak_catalog_aws_apne1`
Enter volume name (default: switch_volume): switch_volume
14:11:39     INFO [d.l.l.deployment.configurator] Found existing volume `switch_volume` in catalog `hinak_catalog_aws_apne1` and schema `switch`
Select a Foundation Model serving endpoint:
[0] [Recommended] databricks-claude-sonnet-4-5
...
Enter a number between 0 and 26: 0
14:11:45     INFO [d.l.l.transpiler.switch_runner] Uploading /Users/hiroyuki.nakazato/0/repos/lakebridge-switch/switch/examples/sql/mssql/input to /Volumes/hinak_catalog_aws_apne1/switch/switch_volume/input-20260116051145-o3rb...
14:11:45     INFO [d.l.l.transpiler.switch_runner] Upload complete: /Volumes/hinak_catalog_aws_apne1/switch/switch_volume/input-20260116051145-o3rb
14:11:45     INFO [d.l.l.transpiler.switch_runner] Triggering Switch job with job_id: 1002520435286464
14:11:46     INFO [d.l.l.transpiler.switch_runner] Switch LLM transpilation job started: https://e2-demo-tokyo.cloud.databricks.com/jobs/1002520435286464/runs/155420908996613

Execution result: Job completed successfully on classic cluster

hiroyukinakazato-db · 2026-01-16T05:38:25Z

@sundarshankar89 @gueniai Updated PR #2222 to use LAKEBRIDGE_CLUSTER_TYPE=CLASSIC environment variable instead of interactive prompt. Ready for review.

sundarshankar89

small nit: this looks good.

src/databricks/labs/lakebridge/cli.py

gueniai

LGTM

hiroyukinakazato-db temporarily deployed to tool January 9, 2026 14:37 — with GitHub Actions Inactive

feat: enhance serverless compute prompt for Switch job installation

5d2adf1

hiroyukinakazato-db temporarily deployed to tool January 13, 2026 00:57 — with GitHub Actions Inactive

docs: update Switch installation docs for compute selection prompt

1381b8c

hiroyukinakazato-db temporarily deployed to tool January 13, 2026 01:08 — with GitHub Actions Inactive

feat: update installation instructions for compute type selection in …

4825029

…Switch

hiroyukinakazato-db temporarily deployed to tool January 13, 2026 01:15 — with GitHub Actions Inactive

Merge branch 'main' into feat/switch-classic-cluster-support

27a3186

hiroyukinakazato-db temporarily deployed to tool January 13, 2026 01:43 — with GitHub Actions Inactive

hiroyukinakazato-db marked this pull request as ready for review January 13, 2026 02:13

hiroyukinakazato-db requested a review from a team as a code owner January 13, 2026 02:13

hiroyukinakazato-db had a problem deploying to tool January 16, 2026 04:45 — with GitHub Actions Error

Merge branch 'main' into feat/switch-classic-cluster-support

22c5eee

hiroyukinakazato-db temporarily deployed to tool January 16, 2026 04:51 — with GitHub Actions Inactive

sundarshankar89 approved these changes Jan 16, 2026

View reviewed changes

src/databricks/labs/lakebridge/cli.py Outdated Show resolved Hide resolved

fix: restore original comment and log message per review feedback

f653402

hiroyukinakazato-db temporarily deployed to tool January 16, 2026 06:29 — with GitHub Actions Inactive

gueniai approved these changes Jan 21, 2026

View reviewed changes

gueniai added this pull request to the merge queue Jan 21, 2026

Merged via the queue into main with commit 7e93fc0 Jan 21, 2026
6 checks passed

gueniai deleted the feat/switch-classic-cluster-support branch January 21, 2026 22:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add classic cluster option for Switch job deployment#2222

feat: add classic cluster option for Switch job deployment#2222
gueniai merged 8 commits intomainfrom
feat/switch-classic-cluster-support

hiroyukinakazato-db commented Jan 9, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Jan 9, 2026 •

edited

Loading

Uh oh!

codecov bot commented Jan 9, 2026 •

edited

Loading

Uh oh!

hiroyukinakazato-db commented Jan 13, 2026

Uh oh!

gueniai commented Jan 14, 2026 •

edited

Loading

Uh oh!

sundarshankar89 commented Jan 15, 2026

Uh oh!

gueniai commented Jan 15, 2026

Uh oh!

hiroyukinakazato-db commented Jan 16, 2026

Uh oh!

hiroyukinakazato-db commented Jan 16, 2026 •

edited

Loading

Uh oh!

hiroyukinakazato-db commented Jan 16, 2026

Uh oh!

sundarshankar89 left a comment

Uh oh!

Uh oh!

gueniai left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

hiroyukinakazato-db commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

What does this PR do?

Relevant implementation details

Usage

Caveats/things to watch out for when reviewing:

Linked issues

Functionality

Tests

Uh oh!

github-actions bot commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

hiroyukinakazato-db commented Jan 13, 2026

Manual Test Evidence

Test 1: Install with serverless compute (default)

Test 2: Install with classic job compute

Test 3: Job execution with classic job compute

Uh oh!

gueniai commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sundarshankar89 commented Jan 15, 2026

Uh oh!

gueniai commented Jan 15, 2026

Uh oh!

hiroyukinakazato-db commented Jan 16, 2026

Uh oh!

hiroyukinakazato-db commented Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Manual Test Evidence (Updated)

Branch and Commit

Test 1: Install with serverless compute (default)

Test 2: Install with classic job compute (using env var)

Test 3: Job execution with classic job compute

Uh oh!

hiroyukinakazato-db commented Jan 16, 2026

Uh oh!

sundarshankar89 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gueniai left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

hiroyukinakazato-db commented Jan 9, 2026 •

edited

Loading

github-actions bot commented Jan 9, 2026 •

edited

Loading

codecov bot commented Jan 9, 2026 •

edited

Loading

gueniai commented Jan 14, 2026 •

edited

Loading

hiroyukinakazato-db commented Jan 16, 2026 •

edited

Loading