tapdb: add implementation of supplycommit.CommitmentTracker and supplycommit.StateMachineStore #1508

Roasbeef · 2025-05-03T04:02:50Z

In this PR, we add a concrete implementation of the two fundamental interfaces that will drive the state machine implemented in #1464.

We start with a state machine on disk for a given group key that also points to the latest supply commitment. When we want to add new elements to the supply tree, we create a new state transition, and then queue pending updates associated with this state transition. Once we've signed and broadcasted the commitment transaction, we then apply the state transition: insert all the updates into the tree, update the state machine to point to that new tree, and finalize the state transition.

See the readme for more details.

ffranr · 2025-05-29T23:03:17Z

tapdb/sqlc/migrations/000036_asset_commit.up.sql

+-- Table storing individual update events associated with a pending transition.
+CREATE TABLE supply_update_events (
+    event_id INTEGER PRIMARY KEY,
+
+    -- Reference to the state transition this event is part of.
+    transition_id BIGINT NOT NULL REFERENCES supply_commit_transitions(transition_id) ON DELETE CASCADE,


an update event can't be persisted unless it references an existing transition, is that correct?
If so, then if we're waiting for tx conf then new update events can't be persisted?

If true, then I don't think it matters a great deal, but from a UX perspective, we will need to return an error to the user along the lines of: unconfirmed supply tx, can't accept new asset ignore right now.

That's a great insight. As of right now, if everything is spun up together and a user attempts to add a new update while we're creating a new commitment on-disk, then this'll fail.

I think we can fix this just by removing the NOT NULL here. We'll need some additional follow ups in the state machine though:

Any time an update comes through in a state, attempt to insert it to disk w/ a nil transition_id.

Once we circle back to the default state, then read all the "dangling" update from disk, and assign them to a state transition (in a single db txn).

Alternatively, we can create some other table, then re-insert them in the default state.

I can either do this in another PR (as it affects this one, and the next), or do the incremental change here. Which is preferred?

guggero

Great stuff! Did a first pass to load some context, will continue with the other PRs in the saga.

tapdb/sqlc/migrations/000036_asset_commit.up.sql

tapdb/sqlc/queries/supply_commit.sql

tapdb/supply_commit.go

guggero · 2025-06-03T19:17:51Z

tapdb/supply_commit.go

+	// NULL (e.g., initial commit before ApplyStateTransition ran), use the
+	// empty root.
+	var rootNode *mssmt.BranchNode
+	if commit.SupplyRootHash == nil || !commit.SupplyRootSum.Valid {


Check length instead of nil?

guggero · 2025-06-03T19:18:59Z

tapdb/supply_commit.go

+// its associated chain confirmation details.
+func fetchCommitment(ctx context.Context, db SupplyCommitStore,
+	commitID sql.NullInt64, groupKeyBytes []byte,
+) (lfn.Option[supplycommit.RootCommitment], lfn.Option[commitmentChainInfo], error) { //nolint:lll


Does the root commitment here really need to be an option? Doesn't look like it's ever None in a non-error case.

I have it as an option as in supplycommit.SupplyStateTransition, it's an option. It's an option here as the very first time we go to create a state transition, OldCommitment won't yet exist.

We can return a value of None in the very fist error case: if we query and it doesn't exist (ErrNoRows), then noneRootCommit with a nil error is returned.

tapdb/supply_commit_test.go

guggero · 2025-06-03T19:26:04Z

tapdb/supply_commit.md

+    }
+```
+
+## State Machine Persistence Logic


Very nice doc!

lightninglabs-deploy · 2025-06-24T02:40:02Z

@Roasbeef, remember to re-request review from reviewers when ready

Roasbeef · 2025-06-25T02:00:32Z

Changed the base to target main.

ffranr · 2025-06-25T23:53:23Z

@Roasbeef here's a rebased version of this PR with additional fixups to address the unit test failures and SQL migration conflict:
https://github.com/ffranr/taproot-assets/commits/wip/1534-ignore-tree/supply-commit-machine/

guggero

LGTM pending existing commits and rebase (and DB migration file re-numbering) 🎉

tapdb/sqlc/migrations/000039_asset_commit.up.sql

tapdb/sqlc/queries/assets.sql

tapdb/supply_commit.go

coveralls · 2025-06-26T22:52:59Z

Pull Request Test Coverage Report for Build 15915388434

Details

766 of 1065 (71.92%) changed or added relevant lines in 4 files are covered.
25 unchanged lines in 7 files lost coverage.
Overall coverage increased (+0.5%) to 39.222%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
universe/interface.go	0	19	0.0%
tapdb/sqlc/supply_commit.sql.go	174	198	87.88%
tapdb/supply_commit.go	590	846	69.74%

Files with Coverage Reduction	New Missed Lines	%
asset/group_key.go	2	62.64%
commitment/tap.go	2	71.14%
internal/test/helpers.go	2	86.95%
rfqmsg/records.go	3	64.11%
asset/asset.go	4	47.09%
internal/test/copy.go	6	82.26%
tapdb/multiverse.go	6	53.28%

Totals
Change from base Build 15899212451:	0.5%
Covered Lines:	30611
Relevant Lines:	78046

💛 - Coveralls

… tree state machine In this commit, we add a new migration for the persistence layer of the supply tree state machine. We track a state machine that points to the latest supply commitment. We update the supply commit via a new state transition, which references a series of updates. Once we're ready to apply, we'll apply the updates, mark the transition as finalized, and finally update the relevant pointers. We also make a change to modify mint_anchor_uni_commitments to point to a supply commitment. This lets us keep track of the set of pre commitments that aren't yet spent.

… new spent_by field

…and supplycommit.StateMachineStore In this commit, we add concrete implementation of both CommitmentTracker and StateMachineStore. This implements the persistence layer for the supply commit state machine. The main method to understand is ApplyStateTransition, as it implements the atomic update of all the various components (tree, state transition, etc) on disk.

Roasbeef added the supply tree label May 3, 2025

levmi assigned Roasbeef May 5, 2025

levmi added this to Taproot-Assets Project Board May 5, 2025

github-project-automation bot moved this to 🆕 New in Taproot-Assets Project Board May 5, 2025

levmi moved this from 🆕 New to 🏗 In progress in Taproot-Assets Project Board May 5, 2025

Roasbeef force-pushed the supply-commit-machine branch from 4778c90 to d718094 Compare May 5, 2025 22:46

Roasbeef force-pushed the universe-supply-trees branch 2 times, most recently from 522c3ee to d786104 Compare May 5, 2025 23:04

Roasbeef force-pushed the supply-commit-machine branch from d718094 to 1db2104 Compare May 5, 2025 23:08

Roasbeef force-pushed the universe-supply-trees branch from d786104 to af5f8f5 Compare May 5, 2025 23:13

Roasbeef force-pushed the supply-commit-machine branch 3 times, most recently from e67b58f to 6b99f70 Compare May 5, 2025 23:54

Roasbeef force-pushed the universe-supply-trees branch from af5f8f5 to 64516c9 Compare May 5, 2025 23:54

Roasbeef force-pushed the supply-commit-machine branch from 6b99f70 to 5056daa Compare May 6, 2025 01:19

Roasbeef force-pushed the universe-supply-trees branch from 64516c9 to 6e0b6a4 Compare May 6, 2025 01:19

Roasbeef force-pushed the supply-commit-machine branch from 5056daa to 5c20025 Compare May 6, 2025 01:27

Roasbeef force-pushed the universe-supply-trees branch from 6e0b6a4 to aa782c8 Compare May 6, 2025 01:27

levmi added this to the v0.7 milestone May 8, 2025

Roasbeef mentioned this pull request May 22, 2025

RPC: Add Ignore Tree Endpoints #1554

Merged

guggero self-requested a review May 28, 2025 06:56

ffranr self-requested a review May 29, 2025 22:25

ffranr reviewed May 29, 2025

View reviewed changes

guggero reviewed Jun 3, 2025

View reviewed changes

Roasbeef force-pushed the universe-supply-trees branch from aa782c8 to 36e3753 Compare June 9, 2025 23:40

Roasbeef force-pushed the supply-commit-machine branch 2 times, most recently from bf61350 to 16ae70c Compare June 10, 2025 00:31

Roasbeef force-pushed the universe-supply-trees branch from 36e3753 to 1c17470 Compare June 10, 2025 00:31

Roasbeef force-pushed the supply-commit-machine branch from 16ae70c to 67728da Compare June 10, 2025 01:12

Roasbeef force-pushed the universe-supply-trees branch from 1c17470 to b5f3d5e Compare June 10, 2025 01:12

Roasbeef force-pushed the universe-supply-trees branch from 8b06b69 to b8988d3 Compare June 21, 2025 01:22

Roasbeef force-pushed the supply-commit-machine branch from 37bdcfa to b5a6c5e Compare June 25, 2025 00:27

Roasbeef force-pushed the universe-supply-trees branch from b8988d3 to 85109d9 Compare June 25, 2025 00:27

Roasbeef force-pushed the supply-commit-machine branch from b5a6c5e to c59b408 Compare June 25, 2025 02:00

Roasbeef changed the base branch from universe-supply-trees to main June 25, 2025 02:00

guggero requested review from guggero and ffranr June 26, 2025 15:45

guggero approved these changes Jun 26, 2025

View reviewed changes

tapdb/sqlc/migrations/000039_asset_commit.up.sql Show resolved Hide resolved

tapdb/sqlc/queries/assets.sql Outdated Show resolved Hide resolved

tapdb/supply_commit.go Outdated Show resolved Hide resolved

Roasbeef force-pushed the supply-commit-machine branch from c59b408 to 99529ae Compare June 26, 2025 22:30

Roasbeef force-pushed the supply-commit-machine branch 2 times, most recently from 87ba694 to e285f19 Compare June 27, 2025 00:20

Roasbeef added 8 commits June 26, 2025 17:35

universe: add Encode+Decode methods for BurnLeaf

6f9a19b

tapdb/sqlc: update mint_anchor_uni_commitments queries to account for…

b8979da

… new spent_by field

tapdb/sqlc: add new supply_commit.sql queries

48e6c15

tapdb/sqlc: generate sqlc structs

4baffdf

tapdb: add tests for SupplyCommitMachine

8b6b2e0

tapdb: add supply_commit.md documentation

277c8f3

Roasbeef force-pushed the supply-commit-machine branch from e285f19 to 277c8f3 Compare June 27, 2025 00:35

ffranr approved these changes Jun 27, 2025

View reviewed changes

github-project-automation bot moved this from 🏗 In progress to 👀 In review in Taproot-Assets Project Board Jun 27, 2025

ffranr added this pull request to the merge queue Jun 27, 2025

Merged via the queue into lightninglabs:main with commit 1b2e06c Jun 27, 2025
18 checks passed

github-project-automation bot moved this from 👀 In review to ✅ Done in Taproot-Assets Project Board Jun 27, 2025

tapdb: add implementation of supplycommit.CommitmentTracker and supplycommit.StateMachineStore #1508

tapdb: add implementation of supplycommit.CommitmentTracker and supplycommit.StateMachineStore #1508

Uh oh!

Conversation

Roasbeef commented May 3, 2025

Uh oh!

ffranr May 29, 2025

Choose a reason for hiding this comment

Uh oh!

Roasbeef Jun 10, 2025

Choose a reason for hiding this comment

Uh oh!

guggero left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

guggero Jun 3, 2025

Choose a reason for hiding this comment

Uh oh!

guggero Jun 3, 2025

Choose a reason for hiding this comment

Uh oh!

Roasbeef Jun 10, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

guggero Jun 3, 2025

Choose a reason for hiding this comment

Uh oh!

lightninglabs-deploy commented Jun 24, 2025

Uh oh!

Roasbeef commented Jun 25, 2025

Uh oh!

ffranr commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

guggero left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

coveralls commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Test Coverage Report for Build 15915388434

Details

💛 - Coveralls

Uh oh!

Uh oh!

Uh oh!

ffranr commented Jun 25, 2025 •

edited

Loading

coveralls commented Jun 26, 2025 •

edited

Loading