Lightweight DDO Compatibility: Spark Piece Indexer

[bajtos]

eta: 2024-09-30

In Filecoin, the main unit for storing user data is Piece, as identified by Piece CID (see Filecoin Spec). On the other hand, the data retrieval process operates at the payload level. The client requests data using a Payload CID and receives back the IPLD DAG of the payload.

The on-chain state, events, and history only provide the PieceCID information about stored data. Retrieval probes like Spark need to map PieceCID to PayloadCIDs to probe for retrievability. There is no straightforward solution for such mapping right now.

In the longer term, we are working with IPNI & Boost projects to provide a way to sample payload block CIDs in a given deal. That project will take several months to complete.

Let's build a lightweight alternative to bridge the time until the proper solution is available, and enable Spark to test DDO deals sooner.

Resources:

• proof of concept: Proof of concept: parse head advertisement piece-indexer#1
• design proposal: doc: initial design proposal piece-indexer#2

Tasks

indexer

• Initial implementation
• Monitoring for whether the indexer is behind or in sync with the chain. Eg track the age of the latest advertisement minus the age of the latest ingested advertisement
• Sentry integration
• Investigate why so many SPs report "cid not found" error

REST API

• Initial implementation
• Sentry integration
• Cloudflare caching proxy