IPIP-0421: HTTP Delegated Routing Reader Privacy Upgrade

src/ipips/ipip-0421.md

@@ -0,0 +1,97 @@
+---
+title: "IPIP-0421: HTTP Delegated Routing Reader Privacy Upgrade"
+date: 2023-05-31
+ipip: proposal
+editors:
+  - name: Andrew Gillis
+    github: gammazero
+  - name: Ivan Schasny
+    github: ischasny 
+  - name: Masih Derkani
+    github: masih
+  - name: Will Scott
+    github: willscott
+order: 421
+tags: ['ipips', 'routing', 'privacy', 'double hashing']
+---
+
+## Summary
+
+This IPIP introduces a HTTP API designed for Privacy Preserving Delegated Content Routing provider lookups. 
+
+## Motivation
+
+Currently, IPFS's privacy safeguards are notably deficient, particularly regarding the Content Routing subsystem. Neither Readers (clients who access files) nor Writers (hosts that store and distribute content) can maintain significant privacy related to the content they produce or consume. Presently, a Content Router or a Passive Observer can discern the identity of a file requested by a client and the specific client making the request during the routing process. This situation allows potential adversaries to gain knowledge about the requested CID. An interested party could then request the same CID and download the corresponding file to track the user's activities. Addressing these privacy concerns has been a long-standing demand from the community.
+
+Recent enhancements to the [IPFS DHT](https://github.com/ipfs/specs/pull/373) and [InterPlanetary Network Indexer (IPNI)](https://github.com/ipni/specs/pull/5) have incorporated Double Hashing to improve Reader Privacy. With Double Hashing, Provider Records become encrypted and non-transparent to Content Routers. Given the original CID, a Content Router can decrypt the relevant Provider Records and supply them through the existing Delegated Routing API. To make use of these privacy enhancements, users must modify their interactions with Content Routers by:
+
+* Utilizing a secondary hash over the original Multihash during content lookup;
+* Decrypting the returned, encrypted Provider Records prior to use; and
+* Optionally retrieving additional encrypted Metadata from the Content Router.
+
+Existing APIs cannot support these changes in interaction, necessitating this IPIP as a step to improve the HTTP Delegated Routing API. This proposal adds new endpoints for delivering encrypted content while maintaining the original API for non-privacy-preserving lookups. Writer Privacy, however, is not within the scope of this IPIP and will be handled separately.
+
+## Detailed design
+
+Please refer to the Delegated Routing Reader Privacy Upgrade specification (:cite[http-routing-reader-privacy-v1]) included with this IPIP for detailed design information.
+
+## Design rationale
+
+The proposed API makes two key changes:
+
+1. It introduces new methods for looking up encrypted Provider Records and encrypted Metadata.
+2. It establishes Hashing and Encryption functions and structures the response payloads.
+
+This proposal does not alter the API's idioms, upholding all data formats, design rationale, and principles established in the original :cite[ipip-0337].
+
+### User benefit
+
+With the proposed APIs, users can protect themselves against malicious actors who might spy on their activities by monitoring their traffic to Content Routers and subsequently downloading identical data. Additionally, this API serves as a first step towards a fully private HTTP Delegated Routing protocol, which would eliminate centralized observers like IPNI routers. 
+
+### Compatibility
+
+#### Backwards Compatibility
+
+Users will need to deliberately activate Reader Privacy on their nodes. A new flag could be introduced into IPFS implementations such as Kubo's HTTP Delegated Content Router configuration to streamline this process. Users on older nodes can continue using the existing API and switch on Reader Privacy later.
+
+Content Routers should maintain the same Quality of Service (QoS) for both Privacy Preserving and regular APIs, as both can be served over the same encrypted data. A shim non-encrypted content router can be implemented to encrypt regular CIDs on the fly, proxy the requests through an encrypted content router and finally decrypt the results before returning them to the user.
+
+It is worth noting that not all Content Routers might adopt Reader Privacy. Default HTTP Delegated Routers like `cid.contact` should have Reader Privacy enabled by default in the latest versions of IPFS implementations such as Kubo and Helia. Users should confirm if their chosen custom router supports Reader Privacy when setting it up.
+
+The `/routing/v1/encrypted/` API will be implemented in existing libraries, such as [`boxo/routing/http`](https://github.com/ipfs/boxo/tree/main/routing/http), and will not introduce any breaking changes to existing clear text endpoints. The API will be introduced in a new minor version.
+
+#### Forward Compatibility
+
+Reader Privacy relies on the use of specific hashing and encryption functions. Altering these functions would require a network-wide migration. Content Routers might not be able to migrate seamlessly, as they do not possess the original values. Such function rotation should occur infrequently and necessitate network-wide efforts. When function rotation is required, the API version will be incremented.
+
+### Security
+
+For details on security, please see the "Threat Modelling" section of :cite[http-routing-reader-privacy-v1].
+
+### Alternatives
+
+When considering alternatives to this IPIP, two potential scenarios and their corresponding technologies are worth exploring:
+
+1. Oblivious HTTP (OHTTP)
+2. Onion Services
+
+In scenario (1), `/routing/v1` would be implemented behind Oblivious HTTP (OHTTP), a protocol proposed by IETF and Cloudflare. OHTTP separates the information about 'who' is making a request from 'what' they are requesting, thereby preventing routing systems such as IPNI instances from viewing both pieces of information concurrently. This would add an additional layer of privacy by obscuring metadata, such as user behavior patterns, IP addresses, and user-agents.
+
+Scenario (2) envisages the `/routing/v1` behind Onion Services. Onion Services provide another approach to concealing the origin of requests by routing them through the Tor network, further enhancing user privacy.
+
+These two scenarios and their corresponding technologies aren't mutually exclusive to this IPIP. Instead, they could be viewed as complementary solutions that could be deployed in conjunction with Double Hashed records, as proposed in this IPIP, to create a more comprehensive privacy solution. The Double Hashing technique encrypts the content of the communication, making it opaque to passive observers. Simultaneously, OHTTP and Onion Services could provide additional privacy layers by obfuscating metadata about who is making a request.
+
+For more information on OHTTP and Onion Services, please refer to these resources:
+
+- [Oblivious HTTP: IETF](https://www.ietf.org/archive/id/draft-thomson-http-oblivious-01.html)
+- [Oblivious HTTP: Cloudflare](https://blog.cloudflare.com/stronger-than-a-promise-proving-oblivious-http-privacy-properties/)
+- [Onion Services](https://community.torproject.org/onion-services/)
+
+### Resources
+
+- [Double-hashed DHT](https://github.com/ipfs/specs/pull/373/)
+- [Reader Privacy in Indexers](https://github.com/ipni/specs/pull/5)
+
+### Copyright
+
+Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).

src/routing/http-routing-reader-privacy-v1.md

@@ -0,0 +1,111 @@
+---
+title: Routing V1 HTTP Delegated Routing Reader Privacy Upgrade
+description: >
+  This specification describes Delegated Routing Reader Privacy Upgrade. It's an
+  incremental improvement to HTTP Delegated Routing API and inherits all of its 
+  formats and design rationale. 
+date: 2023-05-31
+maturity: reliable
+editors:
+  - name: Andrew Gillis
+    github: gammazero
+  - name: Ivan Schasny
+    github: ischasny 
+  - name: Masih Derkani
+    github: masih
+  - name: Will Scott
+    github: willscott
+order: 0
+tags: ['routing', 'double hashing', 'privacy']
+---
+
+This specification describes a new HTTP API for Privacy Preserving Delegated Content Routing provider lookups. It's an extension to HTTP Delegated Routing API and inherits all of its formats and design rationale. 
+
+## API Specification
+
+### Magic Values
+
+ All salts below are 64-bytes long, and represent a string padded with `\x00`.
+
+ - `SALT_DOUBLEHASH = bytes("CR_DOUBLEHASH\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00")`
+ - `SALT_ENCRYPTIONKEY = bytes("CR_ENCRYPTIONKEY\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00")`
+
+ Magic values are needed to calculate different digests from the same value for different purposes. For example a hash of a Multihash that is used for lookups should be different from the one that is used for
+ key derivation, even though both are calculated from the same original value. In order to do that the Multihash is concatenated with different magic values before applying the hash funciton - `SALT_DOUBLEHASH`
+ for lookups and `SALT_ENCRYPTIONKEY` for key derivation as described in the `Glossary`.
+
+### Glossary
+
+- **`enc`** is [AESGCM](https://en.wikipedia.org/wiki/Galois/Counter_Mode) encryption. The following notation will be used for the rest of the specification `enc(passphrase, nonce, payload)`.
+- **`hash`** is [SHA256](https://en.wikipedia.org/wiki/SHA-2) hashing.
+- **`||`** is concatenation of two values.
+- **`deriveKey`** is deriving a 32-byte encryption key from a passphrase that is done as `hash(SALT_ENCRYPTIONKEY || passphrase)`.
+- **`CID`** is the [Content IDentifier](https://github.com/multiformats/cid).
+- **`MH`** is the [Multihash](https://github.com/multiformats/multihash) contained in a `CID`. It corresponds to the 
+digest of a hash function over some content.
+- **`HASH2`** is a second hash over the multihash. Second Hashes must be of `Multihash` format with `DBL_SHA_256` codec. 
+The digest must be calculated as `hash(SALT_DOUBLEHASH || MH)`.
+- **`ProviderRecord`** is a JSON with Provider Record as described in the [HTTP Delegated Routing Specification](http-routing-v1.md).
+- **`ProviderRecordKey`** is a concatentation of `peerID || contextID`. There is no need for explicitly encoding lengths as they are
+already encoded as a part of the multihash format. Max `contextID` length is 64 bytes.
+- **`EncProviderRecordKey`** is `Nonce || enc(deriveKey(multihash), Nonce, ProviderRecordKey)`. Max `EncProviderRecordKey` is 200 bytes. 
+- **`HashProviderRecordKey`**  is a hash over `ProviderRecordKey` that must be calculated as `hash(SALT_DOUBLEHASH || ProviderRecordKey)`.
+- **`Metadata`** is free form bytes that can represent such information such as IPNI metadata. Max `Metadata` length is 1024 bytes. 
+- **`EncMetadata`** is `Nonce || enc(deriveKey(ProviderRecordKey), Nonce, Metadata)`. Max `EncMetadata` length is 2000 bytes.
+
+:::note
+
+Maximum allowed lengths might change without incrementing the API version. Such fields as `contextID` or `Metadata` are free-form bytes and
+their maximum lengths can be changed in the underlying protocols. 
+
+:::
+
+### API
+#### `GET /routing/v1/encrypted/providers/{HASH2}`
+
+##### Response codes
+
+- `200` (OK): the response body contains 1 or more records
+- `404` (Not Found): must be returned if no matching records are found
+- `422` (Unprocessable Entity): request does not conform to schema or semantic constraints
+
+##### Response Body
+
+```json
+{
+    "EncProviderRecordKeys": [
+        "EBxdYDhd.....",
+        "IOknr9DK.....",
+    ]
+}
+```
+
+Where:
+
+- `EncProviderRecordKeys` a list of base64 encoded `EncProviderRecordKey`;
+
+#### `GET /routing/v1/encrypted/metadata/{HashProviderRecordKey}`
+
+##### Response codes
+
+- `200` (OK): the response body contains 1 record
+- `404` (Not Found): must be returned if no matching records are found
+- `422` (Unprocessable Entity): request does not conform to schema or semantic constraints
+
+##### Response Body
+
+```json
+{
+    "EncMetadata": "EBxdYDhd....."
+}
+```
+
+Where:
+
+- `EncMetadatas` is base64 encoded `EncMetadata`;
+
+### Notes
+
+Assembling a full `ProviderRecord` from the encrypted data will require multiple roundtrips to the server. The first one to fetch a list of `EncProviderRecordKey`s and then one per  
+`EncProviderRecordKey` to fetch `EncMetadata`. In order to reduce the number of roundtrips to one the client implementation should use the local libp2p peerstore for multiaddress discovery
+and [libp2p multistream select](https://github.com/multiformats/multistream-select) for protocol negotiation.