Fix serverless compatibility by implementing conditional DataFrame caching

[BesikiML]

🐛 Problem

Reconciliation fails on Databricks serverless compute with:
[NOT_SUPPORTED_WITH_SERVERLESS] PERSIST TABLE is not supported on serverless compute

✅ Solution

Implemented capability-based caching detection that tests if caching actually works:

Changes Made

1. Added is_serverless property - Tests caching by attempting to cache an empty DataFrame
2. Conditional caching - Only caches if is_serverless is False
   • Classic clusters| Spark connect: → DataFrame is cached
   • Serverless: → write to delta

Fixed issue: #1438

Tests

• manually tested on Databricks serverless compute
• manually tested on classic cluster
• added unit tests
• added integration tests

[github-actions]

✅ 131/131 passed, 7 flaky, 5 skipped, 9m29s total

Flaky tests:

• 🤪 test_installs_and_runs_local_bladebridge (19.497s)
• 🤪 test_transpiles_informatica_to_sparksql_non_interactive[True] (16.18s)
• 🤪 test_transpiles_informatica_to_sparksql (17.582s)
• 🤪 test_transpile_teradata_sql (18.596s)
• 🤪 test_transpiles_informatica_to_sparksql_non_interactive[False] (3.066s)
• 🤪 test_transpile_teradata_sql_non_interactive[True] (19.887s)
• 🤪 test_transpile_teradata_sql_non_interactive[False] (4.495s)

_{Running from acceptance #3702}

[codecov]

Codecov Report

❌ Patch coverage is 40.00000% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.54%. Comparing base (2254cbd) to head (28d6a95).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...abricks/labs/lakebridge/reconcile/recon_capture.py	57.14%	3 Missing ⚠️
...bricks/labs/lakebridge/reconcile/reconciliation.py	0.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2218      +/-   ##
==========================================
- Coverage   67.58%   67.54%   -0.04%     
==========================================
  Files          99       99              
  Lines        8825     8835      +10     
  Branches      915      916       +1     
==========================================
+ Hits         5964     5968       +4     
- Misses       2686     2692       +6     
  Partials      175      175              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[m-abulazm]

LGTM

[sundarshankar89]

checking internally if there is a simpler and safer way to check for this. will get back soon.

[sundarshankar89]

LGTM, I don't want to hold this hostage but I would like followup documentation udpate.

That the uc volume should have the same access controls applied to as of data (uc schema where data resides) or should have locked down only to have read and rite access for the app or job user.

cc: @m-abulazm and @BesikiML