books/architecture: port database design document

books/architecture/src/SUMMARY.md

@@ -27,11 +27,37 @@
 
 ---
 
-- [⚪️ Storage](storage/intro.md)
-    - [⚪️ Database abstraction](storage/database-abstraction.md)
-    - [⚪️ Blockchain](storage/blockchain.md)
-    - [⚪️ Transaction pool](storage/transaction-pool.md)
-    - [⚪️ Pruning](storage/pruning.md)
+- [🟢 Storage](storage/intro.md)
+    - [🟢 Database abstraction](storage/db/intro.md)
+        - [🟢 Abstraction](storage/db/abstraction/intro.md)
+            - [🟢 Backend](storage/db/abstraction/backend.md)
+            - [🟢 ConcreteEnv](storage/db/abstraction/concrete_env.md)
+            - [🟢 Trait](storage/db/abstraction/trait.md)
+        - [🟢 Syncing](storage/db/syncing.md)
+        - [🟢 Resizing](storage/db/resizing.md)
+        - [🟢 (De)serialization](storage/db/serde.md)
+        - [🟢 Known issues and tradeoffs](storage/db/issues/intro.md)
+            - [🟢 Abstracting backends](storage/db/issues/traits.md)
+            - [🟢 Hot-swap](storage/db/issues/hot-swap.md)
+            - [🟢 Unaligned bytes](storage/db/issues/unaligned.md)
+            - [🟢 Endianness](storage/db/issues/endian.md)
+            - [🟢 Multimap](storage/db/issues/multimap.md)
+    - [🟢 Common behavior](storage/common/intro.md)
+        - [🟢 Types](storage/common/types.md)
+        - [🟢 `ops`](storage/common/ops.md)
+        - [🟢 `tower::Service`](storage/common/service/intro.md)
+            - [🟢 Initialization](storage/common/service/initialization.md)
+            - [🟢 Requests](storage/common/service/requests.md)
+            - [🟢 Responses](storage/common/service/responses.md)
+            - [🟢 Resizing](storage/common/service/resizing.md)
+            - [🟢 Thread model](storage/common/service/thread-model.md)
+            - [🟢 Shutdown](storage/common/service/shutdown.md)
+    - [🟢 Blockchain](storage/blockchain/intro.md)
+        - [🟢 Schema](storage/blockchain/schema/intro.md)
+            - [🟢 Tables](storage/blockchain/schema/tables.md)
+            - [🟢 Multimap tables](storage/blockchain/schema/multimap.md)
+    - [⚪️ Transaction pool](storage/txpool/intro.md)
+    - [⚪️ Pruning](storage/pruning/intro.md)
 
 ---
 

books/architecture/src/storage/blockchain/intro.md

@@ -0,0 +1,3 @@
+# Blockchain
+This section contains storage information specific to [`cuprate_blockchain`](https://doc.cuprate.org/cuprate_blockchain),
+the database built on-top of [`cuprate_database`](https://doc.cuprate.org/cuprate_database) that stores the blockchain.

books/architecture/src/storage/blockchain/schema/intro.md

@@ -0,0 +1,2 @@
+# Schema
+This section contains the schema of `cuprate_blockchain`'s database tables.

books/architecture/src/storage/blockchain/schema/multimap.md

@@ -0,0 +1,45 @@
+# Multimap tables
+## Outputs
+When referencing outputs, Monero will [use the amount and the amount index](https://github.com/monero-project/monero/blob/c8214782fb2a769c57382a999eaf099691c836e7/src/blockchain_db/lmdb/db_lmdb.cpp#L3447-L3449). This means 2 keys are needed to reach an output.
+
+With LMDB you can set the `DUP_SORT` flag on a table and then set the key/value to:
+```rust
+Key = KEY_PART_1
+```
+```rust
+Value = {
+    KEY_PART_2,
+    VALUE // The actual value we are storing.
+}
+```
+
+Then you can set a custom value sorting function that only takes `KEY_PART_2` into account; this is how `monerod` does it.
+
+This requires that the underlying database supports:
+- multimap tables
+- custom sort functions on values
+- setting a cursor on a specific key/value
+
+## How `cuprate_blockchain` does it
+Another way to implement this is as follows:
+```rust
+Key = { KEY_PART_1, KEY_PART_2 }
+```
+```rust
+Value = VALUE
+```
+
+Then the key type is simply used to look up the value; this is how `cuprate_blockchain` does it
+as [`cuprate_database` does not have a multimap abstraction (yet)](../../db/issues/multimap.md).
+
+For example, the key/value pair for outputs is:
+```rust
+PreRctOutputId => Output
+```
+where `PreRctOutputId` looks like this:
+```rust
+struct PreRctOutputId {
+    amount: u64,
+    amount_index: u64,
+}
+```

books/architecture/src/storage/blockchain/schema/tables.md

@@ -0,0 +1,39 @@
+# Tables
+
+> See also: <https://doc.cuprate.org/cuprate_blockchain/tables> & <https://doc.cuprate.org/cuprate_blockchain/types>.
+
+The `CamelCase` names of the table headers documented here (e.g. `TxIds`) are the actual type name of the table within `cuprate_blockchain`.
+
+Note that words written within `code blocks` mean that it is a real type defined and usable within `cuprate_blockchain`. Other standard types like u64 and type aliases (TxId) are written normally.
+
+Within `cuprate_blockchain::tables`, the below table is essentially defined as-is with [a macro](https://github.com/Cuprate/cuprate/blob/31ce89412aa174fc33754f22c9a6d9ef5ddeda28/database/src/tables.rs#L369-L470).
+
+Many of the data types stored are the same data types, although are different semantically, as such, a map of aliases used and their real data types is also provided below.
+
+| Alias                                              | Real Type |
+|----------------------------------------------------|-----------|
+| BlockHeight, Amount, AmountIndex, TxId, UnlockTime | u64
+| BlockHash, KeyImage, TxHash, PrunableHash          | [u8; 32]
+
+---
+
+| Table              | Key                  | Value                   | Description |
+|--------------------|----------------------|-------------------------|-------------|
+| `BlockHeaderBlobs` | BlockHeight          | `StorableVec<u8>`       | Maps a block's height to a serialized byte form of its header
+| `BlockTxsHashes`   | BlockHeight          | `StorableVec<[u8; 32]>` | Maps a block's height to the block's transaction hashes
+| `BlockHeights`     | BlockHash            | BlockHeight             | Maps a block's hash to its height
+| `BlockInfos`       | BlockHeight          | `BlockInfo`             | Contains metadata of all blocks
+| `KeyImages`        | KeyImage             | ()                      | This table is a set with no value, it stores transaction key images
+| `NumOutputs`       | Amount               | u64                     | Maps an output's amount to the number of outputs with that amount
+| `Outputs`          | `PreRctOutputId`     | `Output`                | This table contains legacy CryptoNote outputs which have clear amounts. This table will not contain an output with 0 amount.
+| `PrunedTxBlobs`    | TxId                 | `StorableVec<u8>`       | Contains pruned transaction blobs (even if the database is not pruned)
+| `PrunableTxBlobs`  | TxId                 | `StorableVec<u8>`       | Contains the prunable part of a transaction
+| `PrunableHashes`   | TxId                 | PrunableHash            | Contains the hash of the prunable part of a transaction
+| `RctOutputs`       | AmountIndex          | `RctOutput`             | Contains RingCT outputs mapped from their global RCT index
+| `TxBlobs`          | TxId                 | `StorableVec<u8>`       | Serialized transaction blobs (bytes)
+| `TxIds`            | TxHash               | TxId                    | Maps a transaction's hash to its index/ID
+| `TxHeights`        | TxId                 | BlockHeight             | Maps a transaction's ID to the height of the block it comes from
+| `TxOutputs`        | TxId                 | `StorableVec<u64>`      | Gives the amount indices of a transaction's outputs
+| `TxUnlockTime`     | TxId                 | UnlockTime              | Stores the unlock time of a transaction (only if it has a non-zero lock time)
+
+<!-- TODO(Boog900): We could split this table again into `RingCT (non-miner) Outputs` and `RingCT (miner) Outputs` as for miner outputs we can store the amount instead of commitment saving 24 bytes per miner output. -->

books/architecture/src/storage/common/intro.md

@@ -0,0 +1,9 @@
+# Common behavior
+The crates that build on-top of the database abstraction ([`cuprate_database`](https://doc.cuprate.org/cuprate_database))
+share some common behavior including but not limited to:
+
+- Defining their specific database tables and types
+- Having an `ops` module
+- Exposing a `tower::Service` API (backed by a threadpool) for public usage
+
+This section provides more details on these behaviors.

books/architecture/src/storage/common/ops.md

@@ -0,0 +1,21 @@
+# `ops`
+Both [`cuprate_blockchain`](https://doc.cuprate.org/cuprate_blockchain)
+and [`cuprate_txpool`](https://doc.cuprate.org/cuprate_txpool) expose an
+`ops` module containing abstracted abstracted Monero-related database operations.
+
+For example, [`cuprate_blockchain::ops::block::add_block`](https://doc.cuprate.org/cuprate_blockchain/ops/block/fn.add_block.html).
+
+These functions build on-top of the database traits and allow for more abstracted database operations.
+
+For example, instead of these signatures:
+```rust
+fn get(_: &Key) -> Value;
+fn put(_: &Key, &Value);
+```
+the `ops` module provides much higher-level signatures like such:
+```rust
+fn add_block(block: &Block) -> Result<_, _>;
+```
+
+Although these functions are exposed, they are not the main API, that would be next section:
+the [`tower::Service`](./service/intro.md) (which uses these functions).

books/architecture/src/storage/common/service/initialization.md

@@ -0,0 +1,9 @@
+# Initialization
+A database service is started simply by calling: [`init()`](https://doc.cuprate.org/cuprate_blockchain/service/fn.init.html).
+
+This function initializes the database, spawns threads, and returns a:
+- Read handle to the database
+- Write handle to the database
+- The database itself
+
+These handles implement the `tower::Service` trait, which allows sending requests and receiving responses `async`hronously.

books/architecture/src/storage/common/service/intro.md

@@ -0,0 +1,65 @@
+# tower::Service
+Both [`cuprate_blockchain`](https://doc.cuprate.org/cuprate_blockchain)
+and [`cuprate_txpool`](https://doc.cuprate.org/cuprate_txpool) provide
+`async` [`tower::Service`](https://docs.rs/tower)s that define database requests/responses.
+
+The main API that other Cuprate crates use.
+
+There are 2 `tower::Service`s:
+1. A read service which is backed by a [`rayon::ThreadPool`](https://docs.rs/rayon)
+1. A write service which spawns a single thread to handle write requests
+
+As this behavior is the same across all users of [`cuprate_database`](https://doc.cuprate.org/cuprate_database),
+it is extracted into its own crate: [`cuprate_database_service`](https://doc.cuprate.org/cuprate_database_service).
+
+## Diagram
+As a recap, here is how this looks to a user of a higher-level database crate,
+`cuprate_blockchain` in this example. Starting from the lowest layer:
+
+1. `cuprate_database` is used to abstract the database
+1. `cuprate_blockchain` builds on-top of that with tables, types, operations
+1. `cuprate_blockchain` exposes a `tower::Service` using `cuprate_database_service`
+1. The user now interfaces with `cuprate_blockchain` with that `tower::Service` in a request/response fashion
+
+```
+                         ┌──────────────────┐
+                         │ cuprate_database │
+                         └────────┬─────────┘
+┌─────────────────────────────────┴─────────────────────────────────┐
+│ cuprate_blockchain                                                │
+│                                                                   │
+│ ┌──────────────────────┐  ┌─────────────────────────────────────┐ │
+│ │ Tables, types        │  │ ops                                 │ │
+│ │ ┌───────────┐┌─────┐ │  │ ┌─────────────┐ ┌──────────┐┌─────┐ │ │
+│ │ │ BlockInfo ││ ... │ ├──┤ │ add_block() │ │ add_tx() ││ ... │ │ │
+│ │ └───────────┘└─────┘ │  │ └─────────────┘ └──────────┘└─────┘ │ │
+│ └──────────────────────┘  └─────┬───────────────────────────────┘ │
+│                                 │                                 │
+│                       ┌─────────┴───────────────────────────────┐ │
+│                       │ tower::Service                          │ │
+│                       │ ┌──────────────────────────────┐┌─────┐ │ │
+│                       │ │ Blockchain{Read,Write}Handle ││ ... │ │ │
+│                       │ └──────────────────────────────┘└─────┘ │ │
+│                       └─────────┬───────────────────────────────┘ │
+│                                 │                                 │
+└─────────────────────────────────┼─────────────────────────────────┘
+                                  │
+		                    ┌─────┴─────┐
+       ┌────────────────────┴────┐ ┌────┴──────────────────────────────────┐
+       │ Database requests       │ │ Database responses                    │
+       │ ┌─────────────────────┐ │ │ ┌───────────────────────────────────┐ │
+       │ │ FindBlock([u8; 32]) │ │ │ │ FindBlock(Option<(Chain, usize)>) │ │
+       │ └─────────────────────┘ │ │ └───────────────────────────────────┘ │
+       │ ┌─────────────────────┐ │ │ ┌───────────────────────────────────┐ │
+       │ │ ChainHeight         │ │ │ │ ChainHeight(usize, [u8; 32])      │ │
+       │ └─────────────────────┘ │ │ └───────────────────────────────────┘ │
+       │ ┌─────────────────────┐ │ │ ┌───────────────────────────────────┐ │
+       │ │ ...                 │ │ │ │ ...                               │ │
+       │ └─────────────────────┘ │ │ └───────────────────────────────────┘ │
+       └─────────────────────────┘ └───────────────────────────────────────┘
+                            ▲          │
+                            │          ▼
+                     ┌─────────────────────────┐
+                     │ cuprate_blockchain user │
+                     └─────────────────────────┘
+```

books/architecture/src/storage/common/service/requests.md

@@ -0,0 +1,8 @@
+# Requests
+Along with the 2 handles, there are 2 types of requests:
+- Read requests, e.g. [`BlockchainReadRequest`](https://doc.cuprate.org/cuprate_types/blockchain/enum.BlockchainReadRequest.html)
+- Write requests, e.g. [`BlockchainWriteRequest`](https://doc.cuprate.org/cuprate_types/blockchain/enum.BlockchainWriteRequest.html)
+
+Quite obviously:
+- Read requests are for retrieving various data from the database
+- Write requests are for writing data to the database

books/architecture/src/storage/common/service/resizing.md

@@ -0,0 +1,15 @@
+# Resizing
+As noted in the [`cuprate_database` resizing section](../../db/resizing.md),
+builders on-top of `cuprate_database` are responsible for resizing the database.
+
+In `cuprate_{blockchain,txpool}`'s case, that means the `tower::Service` must know
+how to resize. This logic is shared between both crates, defined in `cuprate_database_service`:
+<https://github.com/Cuprate/cuprate/blob/0941f68efcd7dfe66124ad0c1934277f47da9090/storage/service/src/service/write.rs#L107-L171>.
+
+By default, this uses a _similar_ algorithm as `monerod`'s:
+
+- [If there's not enough space to fit a write request's data](https://github.com/Cuprate/cuprate/blob/0941f68efcd7dfe66124ad0c1934277f47da9090/storage/service/src/service/write.rs#L130), start a resize
+- Each resize adds around [`1,073,745,920`](https://github.com/Cuprate/cuprate/blob/2ac90420c658663564a71b7ecb52d74f3c2c9d0f/database/src/resize.rs#L104-L160) bytes to the current map size
+- A resize will be [attempted `3` times](https://github.com/Cuprate/cuprate/blob/0941f68efcd7dfe66124ad0c1934277f47da9090/storage/service/src/service/write.rs#L110) before failing
+
+There are other [resizing algorithms](https://doc.cuprate.org/cuprate_database/resize/enum.ResizeAlgorithm.html) that define how the database's memory map grows, although currently the behavior of `monerod` is closely followed (for no particular reason).

books/architecture/src/storage/common/service/responses.md

@@ -0,0 +1,18 @@
+# Responses
+After sending a request using the read/write handle, the value returned is _not_ the response, yet an `async`hronous channel that will eventually return the response:
+```rust,ignore
+// Send a request.
+//                                   tower::Service::call()
+//                                          V
+let response_channel: Channel = read_handle.call(BlockchainReadRequest::ChainHeight)?;
+
+// Await the response.
+let response: BlockchainReadRequest = response_channel.await?;
+```
+
+After `await`ing the returned channel, a `Response` will eventually be returned when
+the `Service` threadpool has fetched the value from the database and sent it off.
+
+Both read/write requests variants match in name with `Response` variants, i.e.
+- `BlockchainReadRequest::ChainHeight` leads to `BlockchainResponse::ChainHeight`
+- `BlockchainWriteRequest::WriteBlock` leads to `BlockchainResponse::WriteBlockOk`

books/architecture/src/storage/common/service/shutdown.md

@@ -0,0 +1,4 @@
+# Shutdown
+Once the read/write handles to the `tower::Service` are `Drop`ed, the backing thread(pool) will gracefully exit, automatically.
+
+Note the writer thread and reader threadpool aren't connected whatsoever; dropping the write handle will make the writer thread exit, however, the reader handle is free to be held onto and can be continued to be read from - and vice-versa for the write handle.