feat: Add centralized manifest storage using relational database - part1 #481

KafkaCat · 2025-09-05T23:02:47Z

Summary

This PR will be the first part of a larger effort to add the centralized manifest metadata service for tonbo. More PR is coming along with way.

What changes are included in this PR?

Add bootstrap.sql to model version snapshot. DDL script is idempotent
Implement current() and update() for ManifestStorage<R> trait that read the latest version snapshot and write a new version snapshot by applying the version edits

Concerns

I believe the centralized manifest metadata service should belongs to a separate repository rather than the public tonbo repo.
I found there is a testcase of VersionEdit::recover() which is specific to reading from version log file. Current data model doesn't support this usecase. I will need to talk to someone to find out whether/how to support it.
Deletion of old, non-active version snapshots data in database are not done yet. This is required to maintain database scale so that query won't took long.

TODOs

Part2:
- Flush deleted sst/wal files into database. Add clean() method to remove actual files and send clean tag
- Add a daemon process to look up database regularly and cleanup unused rows based on active transaction timestamps.
- Implement rewrite() and destroy() interface
Part3:
- Add multi-tenant support: Upon creation of db register self to centralized metadata service.
- Revise data model to include S3 location entity. Upon initialization of db also persist file system/location into database
- Revise query to use db_id to get version snapshot for specific tenant

Are these changes tested?

Run cargo test -- metadata::mds::tests --ignored require-docker

NOTE: The test is actually integration test. We need a better story for running integration test

codecov · 2025-09-07T13:51:40Z

Codecov Report

❌ Patch coverage is 0% with 328 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/metadata/mds.rs	0.00%	308 Missing ⚠️
src/version/mod.rs	0.00%	17 Missing ⚠️
src/version/timestamp.rs	0.00%	3 Missing ⚠️

📢 Thoughts on this report? Let us know!

ethe · 2025-09-07T15:38:01Z

src/metadata/bootstrap.sql

+    scope_id INTEGER REFERENCES scopes(id) ON DELETE RESTRICT,
+    PRIMARY KEY (version_snapshot_id, row_index, col_index),
+
+    CONSTRAINT valid_row_index CHECK (row_index >= 0 AND row_index <= 7), -- harded coded MAX_LEVEL 


the max level num is 6 I think, we hard code leveled compaction as 7 levels, which are level0 to level6, so I think row_index < 7 would be correct

ethe · 2025-09-07T15:40:42Z

src/metadata/mds.rs

+            r#"
+                SELECT id, timestamp, log_length
+                FROM version_snapshots 
+                ORDER BY created_at DESC 


the ordering column is different with https://github.com/tonbo-io/tonbo/pull/481/files#diff-f2ac089b8c9efebe62753973e425342ec1abbb4a4dee365daa6babca6e8a35deR140, it would be clear if we always use the same column to get/set the current timestamp

ethe · 2025-09-07T15:41:14Z

src/metadata/mds.rs

+        // Non recovery mode will persist the log length
+        if !is_recover {
+            version_edits.push(VersionEdit::NewLogLength { len: edit_len });
+            eprint!("I'm here to push: {edit_len}")


return Error here would be better

opps. It was for my testing but apparently I missed this. Will fix

ethe · 2025-09-07T15:44:42Z

sqlx does not support WASM target platform, which means we can not support running Tonbo cloud instance on FaaS based on Deno (it would be better to support this), are there any alternatives?

ethe · 2025-09-07T15:58:42Z

I found there is a testcase of VersionEdit::recover() which is specific to reading from version log file.

IMO recover() should be a behavior of a specific metadata storage implementations, we should not expose this method to storage trait level or beyond, because for remote metadata server, it does not need to call recover and reconstruct the full in-mem version set status, every editting and reading should always request remote service.

What do you think? cc @jonathanc-n @KafkaCat

ethe · 2025-09-07T15:59:33Z

Deletion of old, non-active version snapshots data in database are not done yet. This is required to maintain database scale so that query won't took long.

Yes, this also relates to GC in major compaction.

ethe · 2025-09-07T16:05:29Z

I think remote metadata service implmentation would be better to be an external independent crate (maybe a workspace member in this repository), not inline in Tonbo core, it keeps Tonbo core lean, and lets the backend evolve independently.

ethe · 2025-09-07T16:56:53Z

src/metadata/mds.rs

+        option: Arc<DbOption>,
+    ) -> Result<Self, MetadataServiceError> {
+        let pool = PgPoolOptions::new()
+            .idle_timeout(Some(Duration::from_secs(10 * 60)))


It would be better to add a todo comment here: make it configurable would be better.

ethe · 2025-09-07T16:59:43Z

src/metadata/mds.rs

+                [const { Vec::new() }; MAX_LEVEL],
+                option.clone(),
+                clean_sender.clone(),
+                Arc::new(AtomicU32::new(0)),


Suggested change

Arc::new(AtomicU32::new(0)),

timestamp,

KafkaCat · 2025-09-07T17:13:25Z

I think remote metadata service implmentation would be better to be an external independent crate (maybe a workspace member in this repository), not inline in Tonbo core, it keeps Tonbo core lean, and lets the backend evolve independently.

I can make metadata service into a separate workspace for this PR. But I strongly believe it should belongs to a separate repository to maintain boundary between managed service and core edge use case

KafkaCat · 2025-09-07T17:46:12Z

I found there is a testcase of VersionEdit::recover() which is specific to reading from version log file.

IMO recover() should be a behavior of a specific metadata storage implementations, we should not expose this method to storage trait level or beyond, because for remote metadata server, it does not need to call recover and reconstruct the full in-mem version set status, every editting and reading should always request remote service.

Yeah I think the existing interface is not abstract enough. The interface in my mind is: 1. current(&self, db_id:DbId)->VersionRef<R> 2. update(&self, db_id: DbId, version_edits: Vec<VersionEdits<_>>)->() 3. compact(&self, db_id: DbId) 4. destroy(&self, db_id: DbId)

All implementation should keep an in-memory representation of VersionRef and call update to flush the version edits into persistence. Rewrite is used to compact past set of version edits into a cleaner version (relational model need to revised to only record the version edits delta instead of keeping writing new version every time)

KafkaCat · 2025-09-07T17:51:29Z

sqlx does not support WASM target platform, which means we can not support running Tonbo cloud instance on FaaS based on Deno (it would be better to support this), are there any alternatives?

Good question! Sqlx is a widely recognized crate for data access layer so I just use it. I haven't thought of WASM compatibility. Let's talk offline on usecase of supporting javascript runtime. I think there is some knowledge gap for me to fill.

~~Meanwhile let me research if there is any other crate support WASM target~~ If I switch to SQLite eco-system then there is diesel ORM with sqlite wasm support It also makes testing story a little easier since sqlite crate can spawn in-memory instance. Maybe that's the approach forward if WASM support is a hard requirement.

KafkaCat · 2025-09-08T04:32:03Z

Synced with ethe@ offline. Notes:

We will start a new repository and move this implementation to there to keep core tonbo clean
I will define a new ManifestStorage interface that works for both log-based and db-based approach
I will investigate the change log approach that only persists the version edits delta and keep an in-memory presentation of Version. This way it's fast to read from database
I will also migrate to sqlite eco-system to better support wasm use case

KafkaCat and others added 3 commits September 1, 2025 22:58

wip

425a806

src complete; test broken

9885447

add test using testcontainers

6ca4857

jonathanc-n requested a review from ethe September 5, 2025 23:03

jonathanc-n self-assigned this Sep 5, 2025

ethe requested a review from jonathanc-n September 6, 2025 04:54

ethe assigned KafkaCat Sep 6, 2025

jonathanc-n removed their assignment Sep 6, 2025

remove unused crate and fix cargo format

a02ad7e

ethe reviewed Sep 7, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: Add centralized manifest storage using relational database - part1 #481

feat: Add centralized manifest storage using relational database - part1 #481

Uh oh!

KafkaCat commented Sep 5, 2025 •

edited

Loading

Uh oh!

codecov bot commented Sep 7, 2025

Uh oh!

ethe Sep 7, 2025 •

edited

Loading

Uh oh!

ethe Sep 7, 2025

Uh oh!

ethe Sep 7, 2025

Uh oh!

KafkaCat Sep 7, 2025

Uh oh!

ethe commented Sep 7, 2025

Uh oh!

ethe commented Sep 7, 2025

Uh oh!

ethe commented Sep 7, 2025

Uh oh!

ethe commented Sep 7, 2025

Uh oh!

ethe Sep 7, 2025

Uh oh!

ethe Sep 7, 2025

Uh oh!

KafkaCat commented Sep 7, 2025

Uh oh!

KafkaCat commented Sep 7, 2025 •

edited

Loading

Uh oh!

KafkaCat commented Sep 7, 2025 •

edited

Loading

Uh oh!

KafkaCat commented Sep 8, 2025

Uh oh!

Uh oh!

feat: Add centralized manifest storage using relational database - part1 #481

Are you sure you want to change the base?

feat: Add centralized manifest storage using relational database - part1 #481

Uh oh!

Conversation

KafkaCat commented Sep 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What changes are included in this PR?

Concerns

TODOs

Are these changes tested?

Uh oh!

codecov bot commented Sep 7, 2025

Codecov Report

Uh oh!

ethe Sep 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ethe Sep 7, 2025

Choose a reason for hiding this comment

Uh oh!

ethe Sep 7, 2025

Choose a reason for hiding this comment

Uh oh!

KafkaCat Sep 7, 2025

Choose a reason for hiding this comment

Uh oh!

ethe commented Sep 7, 2025

Uh oh!

ethe commented Sep 7, 2025

Uh oh!

ethe commented Sep 7, 2025

Uh oh!

ethe commented Sep 7, 2025

Uh oh!

ethe Sep 7, 2025

Choose a reason for hiding this comment

Uh oh!

ethe Sep 7, 2025

Choose a reason for hiding this comment

Uh oh!

KafkaCat commented Sep 7, 2025

Uh oh!

KafkaCat commented Sep 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

KafkaCat commented Sep 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

KafkaCat commented Sep 8, 2025

Uh oh!

Uh oh!

KafkaCat commented Sep 5, 2025 •

edited

Loading

ethe Sep 7, 2025 •

edited

Loading

KafkaCat commented Sep 7, 2025 •

edited

Loading

KafkaCat commented Sep 7, 2025 •

edited

Loading