doc: initial design proposal #2

bajtos · 2024-07-04T11:52:48Z

chore: setup Prettier formatting for Markdown
docs: initial design proposal

See also #1 for a PoC fetching & parsing a single advertisement.

Links:

Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>

patrickwoodhead · 2024-07-17T19:50:59Z

docs/design.md

+
+Pieces are immutable. If we receive an advertisement saying that a payload block
+CID was found in a piece CID, then this information remains valid forever, even
+after the SP advertise that they are no longer storing that block. This means


But if the SP says that they are no longer storing the piece then we should no longer make retrieval requests to that SP for any payloads in the piece, no?

Yes, we should not make any retrievals for such payload.

I am envisioning the following architecture:

A piece indexer I describe in this document. It will be eventually replaced by a proper IPNI reverse index solution.

A deal tracker - a component that listens for actor events and builds a list of active deals - conceptually a list of pairs (piece_cid, miner_id). We will need to build this.

When Spark build a list of tasks for the current round, it will ask the deal tracker for 1000 active deals. This ensures we test retrieval for active deals only.

When Spark checker tests retrieval, it will first consult the piece indexer to convert deal's piece_cid to a payload CID to retrieve.

It's ok if the piece indexer stores data for expired deals, because Spark is not going to ask for that data.

Of course, storing expired deals unnecessarily increases storage requirements, but since we want to run this service only for 2-6 months, I think it's fine.

Sgtm, could you add this context to the document please? I was also not aware of the distinction between the piece indexer and the deal tracker

docs/design.md

patrickwoodhead · 2024-07-17T19:54:07Z

docs/design.md

+advertised for the same Payload CID. Different SPs can advertise different lists
+(e.g. the entries can be ordered differently) or can even cheat and submit CIDs
+that are not part of the Piece. Our indexer must scope the information to each
+index provider.


By index proivder do you mean IPNI instance or Storage provider creating indexes/advertisements?

The index provider is the actor submitting data to the index. Typically, Boost and Venus Droplet.

@patrickwoodhead
How can I improve the text to make this easier to understand?

Is "index provider" the term to use? If so, I don't think this needs to be clearer up. Maybe we could add the term definition to a definition section in this document?

The term "index provider" is used by IPNI:

https://github.com/ipni/index-provider

https://github.com/MarcoPolo/http-index-provider-example

However, the spec uses just "provider" 🤷🏻‍♂️

https://github.com/ipni/specs/blob/90648bca4749ef912b2d18f221514bc26b5bef0a/IPNI.md#terminology

Publisher: This is an entity that publishes advertisements and index data to an indexer. It is usually, but not always, the same as the data provider. A publisher is identified by a libp2p peer ID.

I like the idea of adding a section to explain the terminology 👍🏻

patrickwoodhead · 2024-07-17T19:57:31Z

docs/design.md

+
+The response provides all the metadata we need to download the advertisements:
+
+- `Publisher.Addrs` describes where we can contact SP's index provider


what does this mean?

I believe it's the HTTP address where the SP can respond to indexer requests

I think it doesn't have to be an HTTP address; it could be a Graphsync/libp2p address, too.

However, there is a push from IPNI to move to HTTP transport. For our lighweight indexer, we will require SPs to support the HTTP protocol for handling indexer requests.

Unfortunately, this requires SPs to tweak their Boost configuration and provide the public hostname at which Boost can be reached from outside. On the bright side, I think it's most likely that SPs have to configure this option anyways if they want cid.contact to receive their advertisements, in which case our lightweight indexer is not adding any new requirements.

https://boost.filecoin.io/configuration/http-indexer-announcement

I reworded this item as follows:

Publisher.Addrs describes where we can contact SP's index provider to retrieve content for CIDs, e.g., advertisements.

patrickwoodhead · 2024-07-17T19:58:04Z

docs/design.md

+The response provides all the metadata we need to download the advertisements:
+
+- `Publisher.Addrs` describes where we can contact SP's index provider
+- `LastAdvertisement` contains the CID of the head advertisement


Is this the Last advertisement for a specific SP or just in the IPNI instance altogether?

It's the last advertisement for a specific SP.

All fields described in this section are per-provider.

How can I improve the text to make this more clear?

What about

The response provides all the metadata we need to download the advertisements:

->

The response provides all the per-SP metadata we need to download the advertisements:

Reworded as follows:

The response provides all the metadata we need to download the advertisements. For each index provider, the response includes: - `Publisher.Addrs` describes where we can contact SP's index provider to retrieve content for CIDs, e.g., advertisements. - `LastAdvertisement` contains the CID of the head advertisement from the SP

patrickwoodhead · 2024-07-17T20:00:14Z

docs/design.md

+  advertisements from `last_head` to the end of the chain were already
+  processed.
+
+- `next_head` - The CID of the most recent head seen by cid.contact. This is


how does next_head differ from head? Do we not start each walk from the current head?

The initial walk will take a long time to complete. While we are walking the "old" chain, new advertisements (new heads) will be announced to IPNI.

next_head is the latest head announced to IPNI

head is the advertisement where the current walk-in-progress started

I suppose we don't need to keep track of next_head. When the current walk finishes, we will wait up to one minute until we make another request to cid.contact to find what are the latest heads for each SPs.

In my current proposal, when the current walk finishes, we can immediately continue with walking from the next_head.

See also the diagram in https://github.com/filecoin-station/piece-indexer/pull/2/files#r1689884939

I captured my explanation in the design doc.

patrickwoodhead · 2024-07-17T20:03:28Z

docs/design.md

+- `tail` - The CID of the next advertisement in the chain that we need to
+  process in the current walk.
+
+Every minute, fetch the latest providers from cid.contact. For each provider


Each minute, we are running an algorithm that has a complexity of the order of the number of providers. Do we have any concerns about how much processing we will be doing each minute? Might this process take more than a minute to run?

What about with one minute delay between every run?

I think this should be fine. Every minute, we need to do these ~~three~~ four steps:

Run a SQL query to get the next_head for each SP

Make one HTTP call to cid.contact to find the latest advertisement heads announced by all SPs

Run one SQL query to update next_head for all SPs where there is a new head

Kick-off advertisement walks (up to one walk per provider). These walks are executed in the background and don't block this loop.

I updated the spec to capture my explanation.

patrickwoodhead · 2024-07-17T20:06:07Z

docs/design.md

+spark-evaluate to verify the authenticity of results reported by checker nodes:
+
+```
+GET /sample/{providerId}/{pieceCid}?seed={seed}


can we add an endpoint that allows a SP to see which records there are in the DB associated to them? As per F8 Ptrk's comment a week or so back

See the "Observability" section below. Do you think we need something different?

docs/design.md

juliangruber · 2024-07-24T09:01:51Z

docs/design.md

+- `tail` - The CID of the next advertisement in the chain that we need to
+  process in the current walk.
+
+Every minute, fetch the latest providers from cid.contact. For each provider


What about with one minute delay between every run?

docs/design.md

juliangruber · 2024-07-24T09:05:06Z

docs/design.md

+  where we need to start the next walk from.
+
+- `head` - The CID of the head advertisement we started the current walk from.
+  We update this value whenever we start a new walk.


Why do we need this state? Should it not be enough to go from next_head to last_head and afterwards update last_head?

Or is this for the case where this takes a long time and we want to be able to resume the chain walk? In this case, what about we simplify the design by saying that we will only ever walk X links per iteration, thereby eliminating the need for the head state?

Then we could also remove tail

The current walk starts from head and walks up to last_head. When the current walk reaches last_head, we need to set last_head ← head so that the next walk knows where to stop.

next_head is updated every minute when we query cid.contact for the latest heads. If the walk takes longer than a minute to finish, then next_head will change and we cannot use it for last_head.

What we can do, is to remove next_head, as I explained in https://github.com/filecoin-station/piece-indexer/pull/2/files#r1689865074

In this case, what about we simplify the design by saying that we will only ever walk X links per iteration, thereby eliminating the need for the head state?

We must always walk the chain all the way to the genesis or to the entry we have already seen & processed. Here is how the state looks like in the middle of a walk:

next_head ↓ (entries announced after we started the current walk) ↓ head ↓ (entries visited in this walk) ↓ tail ↓ (entries NOT visited yet) ↓ last_head ↓ (entries visited in the previous walks) ↓ (null)

docs/design.md

Co-authored-by: Julian Gruber <julian@juliangruber.com>

Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>

bajtos added 2 commits July 4, 2024 10:25

chore: setup Prettier formatting for Markdown

2e0b440

Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>

docs: initial design proposal

31a86c3

Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>

bajtos requested review from juliangruber and patrickwoodhead July 4, 2024 11:52

patrickwoodhead reviewed Jul 17, 2024

View reviewed changes

juliangruber requested changes Jul 24, 2024

View reviewed changes

bajtos and others added 10 commits July 24, 2024 15:28

Update docs/design.md

c2910e9

Co-authored-by: Julian Gruber <julian@juliangruber.com>

Update docs/design.md

8c993b2

Co-authored-by: Julian Gruber <julian@juliangruber.com>

Update docs/design.md

75b7480

Co-authored-by: Julian Gruber <julian@juliangruber.com>

Update docs/design.md

4864eea

Co-authored-by: Julian Gruber <julian@juliangruber.com>

Update docs/design.md

320ef5e

Co-authored-by: Julian Gruber <julian@juliangruber.com>

Update docs/design.md

7c1e28b

Update docs/design.md

642a972

Co-authored-by: Julian Gruber <julian@juliangruber.com>

clarify the algorithm

3bbc2c2

Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>

fix formatting

9349c62

Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>

clarify Publisher.Addrs

24c060c

Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>

bajtos mentioned this pull request Jul 26, 2024

Lightweight DDO Compatibility: Spark Piece Indexer CheckerNetwork/roadmap#143

Closed

6 tasks

bajtos added 2 commits September 4, 2024 08:38

fixup! address review comments

21af1f6

Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>

fixup! more cleanup

560c65e

Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>

bajtos merged commit 0fc5efd into main Sep 4, 2024

bajtos deleted the design-doc branch September 4, 2024 08:03


		The response provides all the metadata we need to download the advertisements:

		- `Publisher.Addrs` describes where we can contact SP's index provider

doc: initial design proposal #2

doc: initial design proposal #2

Uh oh!

Conversation

bajtos commented Jul 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bajtos Jul 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bajtos Jul 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bajtos commented Jul 4, 2024 •

edited

Loading

bajtos Jul 25, 2024 •

edited

Loading

bajtos Jul 24, 2024 •

edited

Loading