Refactor recon intermediate persistence

[m-abulazm]

Changes

What does this PR do?

• Clean checkpoint volume after reconcile runs
• Remove usage of overwrite write mode
• use delta volumes if running on databricks instead of parquet
• Refactor interface to hide implementation details and be more generic
• add TODO to mark where we need to persist to delta instead of hitting source system

Caveats/things to watch out for when reviewing:

On Serverless we cannot use cache/persist, so we use Delta writes acting as materialization boundaries

Linked issues

Resolves #1056
Advances #1905, #1438

Functionality

• added relevant user documentation
• added new CLI command
• modified existing command: databricks labs lakebridge ...
• ... +add your own

Tests

• manually tested
• added unit tests
• added integration tests

[codecov]

Codecov Report

❌ Patch coverage is 0% with 85 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.82%. Comparing base (cff1b55) to head (829d53b).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...abricks/labs/lakebridge/reconcile/recon_capture.py	0.00%	43 Missing ⚠️
...ridge/reconcile/trigger_recon_aggregate_service.py	0.00%	25 Missing ⚠️
...labs/lakebridge/reconcile/trigger_recon_service.py	0.00%	10 Missing ⚠️
...rc/databricks/labs/lakebridge/reconcile/compare.py	0.00%	4 Missing ⚠️
...bricks/labs/lakebridge/reconcile/reconciliation.py	0.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2230      +/-   ##
==========================================
- Coverage   63.95%   63.82%   -0.13%     
==========================================
  Files          99       99              
  Lines        8644     8661      +17     
  Branches      890      890              
==========================================
  Hits         5528     5528              
- Misses       2944     2961      +17     
  Partials      172      172              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

✅ 130/130 passed, 9 flaky, 5 skipped, 30m37s total

Flaky tests:

• 🤪 test_installs_and_runs_local_bladebridge (21.792s)
• 🤪 test_installs_and_runs_pypi_bladebridge (29.594s)
• 🤪 test_transpiles_informatica_to_sparksql_non_interactive[False] (17.51s)
• 🤪 test_transpiles_informatica_to_sparksql_non_interactive[True] (17.451s)
• 🤪 test_transpiles_informatica_to_sparksql (18.495s)
• 🤪 test_transpile_teradata_sql_non_interactive[True] (21.975s)
• 🤪 test_transpile_teradata_sql (6.64s)
• 🤪 test_transpile_teradata_sql_non_interactive[False] (5.896s)
• 🤪 test_recon_databricks_job_succeeds (20m46.509s)

_{Running from acceptance #3546}

[BesikiML]

LGTM

[m-abulazm]

Followup issue created to tackle the TODOs #2257

[sundarshankar89]

LGTM, I will add a FAQ about retaining the governance model when reconciling PII information, so any failures to clean up to uc volume shouldn't result in any unintended consiquence