"borrow checker invariants" section of the "leveraging the type system" chapter

src/SUMMARY.md

@@ -445,6 +445,11 @@
       - [Serializer: implement Struct](idiomatic/leveraging-the-type-system/typestate-pattern/typestate-generics/struct.md)
       - [Serializer: implement Property](idiomatic/leveraging-the-type-system/typestate-pattern/typestate-generics/property.md)
       - [Serializer: Complete implementation](idiomatic/leveraging-the-type-system/typestate-pattern/typestate-generics/complete.md)
+  - [Borrow checking invariants](idiomatic/leveraging-the-type-system/borrow-checker-invariants.md)
+    - [Generalizing "Ownership"](idiomatic/leveraging-the-type-system/borrow-checker-invariants/generalizing-ownership.md)
+    - [Single-use values](idiomatic/leveraging-the-type-system/borrow-checker-invariants/single-use-values.md)
+    - [Aliasing XOR Mutability](idiomatic/leveraging-the-type-system/borrow-checker-invariants/aliasing-xor-mutability.md)
+    - [Lifetime Relationships & External Resources](idiomatic/leveraging-the-type-system/borrow-checker-invariants/lifetime-connections.md)
 
 ---
 

src/idiomatic/leveraging-the-type-system/borrow-checker-invariants.md

@@ -0,0 +1,80 @@
+---
+minutes: 10
+---
+
+# Using the Borrow checker to enforce Invariants
+
+The logic of the borrow checker, while tied to "memory ownership", can be
+abstracted away from this central use case to model other problems and prevent
+API misuse.
+
+```rust,editable
+// Doors can be open or closed, and you need the right key to lock or unlock
+// one. Modelled with a Shared key and Owned door.
+pub struct DoorKey {
+    pub key_shape: u32,
+}
+pub struct LockedDoor {
+    lock_shape: u32,
+}
+pub struct OpenDoor {
+    lock_shape: u32,
+}
+
+fn open_door(key: &DoorKey, door: LockedDoor) -> Result<OpenDoor, LockedDoor> {
+    if door.lock_shape == key.key_shape {
+        Ok(OpenDoor { lock_shape: door.lock_shape })
+    } else {
+        Err(door)
+    }
+}
+
+fn close_door(key: &DoorKey, door: OpenDoor) -> Result<LockedDoor, OpenDoor> {
+    if door.lock_shape == key.key_shape {
+        Ok(LockedDoor { lock_shape: door.lock_shape })
+    } else {
+        Err(door)
+    }
+}
+
+fn main() {
+    let key = DoorKey { key_shape: 7 };
+    let closed_door = LockedDoor { lock_shape: 7 };
+    let opened_door = open_door(&key, closed_door);
+    if let Ok(opened_door) = opened_door {
+        println!("Opened the door with key shape '{}'", key.key_shape);
+    } else {
+        eprintln!(
+            "Door wasn't opened! Your key only opens locks with shape '{}'",
+            key.key_shape
+        );
+    }
+}
+```
+
+<details>
+
+<!-- TODO: link to typestate when that gets merged. -->
+
+- The borrow checker has been used to prevent use-after-free and multiple
+  mutable references up until this point, and we've used types to shape and
+  restrict use of APIs already using the "typestate" pattern.
+
+- This example uses the ownership & borrowing rules to model the locking and
+  unlocking of a door. We can try to open a door with a key, but if it's the
+  wrong key the door is still closed (here represented as an error) and the key
+  persists regardless.
+
+- The rules of the borrow checker exist to prevent developers from accessing,
+  changing, and holding onto data in memory in unpredictable ways without being
+  so restrictive that it would prevent _writing software_. The underlying
+  logical system does not "know" what memory is. All it does is enforce a
+  specific set of rules of how different operations affect what later operations
+  are possible.
+
+- Those rules can apply to many other cases: We can piggy-back onto the rules of
+  the borrow checker to design APIs to be harder or impossible to misuse, even
+  when there's little or no "memory safety" concerns in the problem domain. This
+  section will walk through some of those different domains.
+
+</details>

src/idiomatic/leveraging-the-type-system/borrow-checker-invariants/aliasing-xor-mutability.md

@@ -0,0 +1,99 @@
+---
+minutes: 15
+---
+
+# Mutually Exclusive References, or "Aliasing XOR Mutability"
+
+We can use the mutual exclusion of `&T` and `&mut T` references for a single
+value to model some constraints.
+
+```rust,editable,compile_fail
+pub struct Transaction(/* specifics omitted */);
+pub struct QueryResult(String);
+
+pub struct DatabaseConnection {
+    transaction: Transaction,
+    query_results: Vec<QueryResult>,
+}
+
+impl DatabaseConnection {
+    pub fn new() -> Self {
+        Self {
+            transaction: Transaction(/* again, specifics omitted */),
+            query_results: vec![],
+        }
+    }
+    pub fn get_transaction(&mut self) -> &mut Transaction {
+        &mut self.transaction
+    }
+    pub fn results(&self) -> &[QueryResult] {
+        &self.query_results
+    }
+    pub fn commit(&mut self) {
+        /* Work omitted, including sending/clearing the transaction */
+        println!("Transaction committed!")
+    }
+}
+
+pub fn do_something_with_transaction(transaction: &mut Transaction) {}
+
+fn main() {
+    let mut db = DatabaseConnection::new();
+    let mut transaction = db.get_transaction();
+    do_something_with_transaction(transaction);
+    let assumed_the_transactions_happened_immediately = db.results(); // ❌🔨
+    do_something_with_transaction(transaction);
+    // Works, as the lifetime of "transaction" as a reference ended above.
+    let assumed_the_transactions_happened_immediately_again = db.results();
+    db.commit();
+}
+```
+
+<details>
+
+- Aliasing XOR Mutability means "we can have multiple immutable references, a
+  single mutable reference, but not both."
+
+- This example shows how we can use the mutual exclusion of these kinds of
+  references to dissuade a user from reading query results while using a
+  transaction API.
+
+  This might happen if the user is working under the false assumption that the
+  queries being written to the transaction happen "immediately" rather than
+  being queued up and performed together.
+
+- By borrowing one field of a struct via a method that returns a mutable /
+  exclusive reference we prevent access to the other fields of that struct under
+  a shared / non-exclusive reference until the lifetime of that borrow ends.
+
+  Note: This has to be via a method, as the compiler can reason about borrowing
+  different fields in mutable/shared ways simultaneously if that borrowing is
+  done manually.
+
+  Demonstrate:
+
+  - Change the instances of `db.get_transaction()` and `db.results()` to manual
+    borrows (`&mut db.transaction` and `&db.query_results` respectively) to show
+    the difference in what the borrow checker allows.
+
+  - Put the non-`main` part of this example in a module to reiterate that this
+    manual access is not possible across module boundaries.
+
+- As laid out in [generalizing ownership](generalizing-ownership.md) we can look
+  at the ways Mutable References and Shareable References interact to see if
+  they fit with the invariants we want to uphold for an API.
+
+- In this case, having the query results not public and placed behind a getter
+  function, we can enforce the invariant "users of this API are not looking at
+  the query results at the same time as they are writing to a transaction."
+
+- The "don't look at query results while building a transaction" invariant can
+  still be circumvented, how so?
+
+  - The user could access the transaction solely through `db.get_transaction()`,
+    leaving the lifetime too temporary to prevent access to `db.results()`.
+
+  - How could we avoid this by working in other concepts from "Leveraging the
+    Type System"?
+
+</details>

src/idiomatic/leveraging-the-type-system/borrow-checker-invariants/generalizing-ownership.md

@@ -0,0 +1,88 @@
+---
+minutes: 15
+---
+
+# Generalizing Ownership
+
+The logic of the borrow checker, while modelled off "memory ownership", can be
+abstracted away from that use case to model other problems where we want to
+prevent API misuse.
+
+```rust,editable,compile_fail
+// An internal data type to have something to hold onto.
+pub struct Internal;
+// The "outer" data.
+pub struct Data(Internal);
+
+fn shared_use(value: &Data) -> &Internal {
+    &value.0
+}
+fn exclusive_use(value: &mut Data) -> &mut Internal {
+    &mut value.0
+}
+fn deny_future_use(value: Data) {}
+
+fn main() {
+    let mut value = Data(Internal);
+    let deny_mut = shared_use(&value);
+    let try_to_deny_immutable = exclusive_use(&mut value); // ❌🔨
+    let more_mut_denial = &deny_mut;
+    deny_future_use(value);
+    let even_more_mut_denial = shared_use(&value); // ❌🔨
+}
+```
+
+<details>
+
+- This example re-frames the borrow checker rules away from references and
+  towards semantic meaning in non-memory-safety settings. Nothing is being
+  mutated, nothing is being sent across threads.
+
+- When a new feature is introduced to users, it is often done so with a specific
+  idea of what it will be used for.
+
+  Over time, users may develop ways of using that feature in ways that may have
+  not been foreseen.
+
+  In 2004, Java 5 introduced Generics with the
+  [main stated purpose of enabling type safe collections](https://jcp.org/en/jsr/detail?id=14).
+
+  Since then, users and developers of the language expanded the use of generics
+  to other areas of type safe API design.
+  <!-- TODO: Reference how this was adopted -->
+
+  What we aim to do here is similar: The borrow checker, after being introduced
+  to people with the purpose of avoiding use-after-free and data races, is being
+  used to model things that have nothing to do with preventing those classes of
+  misuse.
+
+- To use the borrow checker as a problem solving tool, we will need to "forget"
+  that the original purpose of it is to prevent mutable aliasing in the context
+  of preventing use-after-frees and data races, instead imagining and working
+  within situations where the rules are the same but the meaning is slightly
+  different.
+
+- In rust's borrow checker we have access to three different ways of "taking" a
+  value:
+
+  <!-- TODO: actually link to the RAII section when it has been merged. -->
+  - Owned value `T`. Very permissive case, to the point where mutability can be
+    re-set, but demands that nothing else is using it in any context and drops
+    the value when scope ends (unless that scope returns this value) (see:
+    RAII.)
+
+  - Mutable Reference `&mut T`. While holding onto a mutable reference we can
+    still "dispatch" to methods and functions that take an immutable, shared
+    reference of the value but only as long as we're not aliasing immutable,
+    shared references to related data "after" that dispatch.
+
+  - Shared Reference `&T`. Allows aliasing but prevents mutable access while any
+    of these exist. We can't "dispatch" to methods and functions that take
+    mutable references when all we have is a shared reference.
+
+- Remember that every `&T` and `&mut T` has a lifetime, just one the user
+  doesn't have to annotate or think about most of the time. We get to avoid
+  annotating a lot of lifetimes because the rust compiler can elide the majority
+  of them. See: [Lifetime Elision](../../../lifetimes/lifetime-elision.md).
+
+</details>

src/idiomatic/leveraging-the-type-system/borrow-checker-invariants/lifetime-connections.md

@@ -0,0 +1,103 @@
+---
+minutes: 20
+---
+
+# Lifetime "Connections" & External Resources
+
+Using `PhantomData` in conjunction with lifetimes lets us say "this value may
+own its data, but it can only live as long as the value that generated it" in
+rust's type system.
+
+```rust,editable,compile_fail
+use std::marker::PhantomData;
+pub struct Tag;
+pub struct ErasedData<'a> {
+    data: String,
+    _phantom: PhantomData<&'a ()>,
+}
+impl<'a> ErasedData<'a> {
+    pub fn get(&self) -> &str {
+        &self.data
+    }
+}
+pub struct TaggedData<T> {
+    data: String,
+    _phantom: PhantomData<T>,
+}
+impl<T> TaggedData<T> {
+    pub fn new(data: String) -> Self {
+        Self { data, _phantom: PhantomData }
+    }
+    pub fn consume(self) {}
+    pub fn get_erased(&self) -> ErasedData<'_> {
+        // has an owned String, but _phantom holds onto the lifetime of the
+        // TaggedData that created it.
+        ErasedData { data: self.data.clone(), _phantom: PhantomData }
+    }
+}
+
+fn main() {
+    let tagged_data: TaggedData<Tag> = TaggedData::new("Real Data".to_owned());
+    // Get the erased-but-still-linked data.
+    let erased_owned_and_linked = tagged_data.get_erased();
+    tagged_data.consume();
+    // Owned by `erased_owned_and_linked` but still connected to `tagged_data`.
+    println!("{}", erased_owned_and_linked.get()); // ❌🔨
+}
+```
+
+<details>
+
+- `PhantomData` lets developers "tag" types with type and lifetime parameters
+  that are not "really" present in the struct or enum.
+
+  `PhantomData` can be used with the Typestate pattern to have data with the
+  same structure i.e. `TaggedData<Start>` can have methods or trait
+  implementations that `TaggedData<End>` doesn't.
+
+  It can also be used to encode a connection between the lifetime of one value
+  and another, while both values still maintain separate owned data within them.
+
+- This is really useful for modelling a bunch of relationships between data,
+  where we want to establish that while a type has owned values within it is
+  still connected to another piece of data and can only live as long as it.
+
+  Consider a case where you want to return owned data from a method, but you
+  don't want that data to live longer than the value that created it.
+
+- Lifetimes need to come from somewhere! We can't build functions of the form
+  `fn lifetime_shenanigans<'a>(owned: OwnedData) -> &'b Data` (without tying
+  `'b` to `'a` in some way).
+
+  Lifetime elision hides where a lot of lifetimes come from, but that doesn't
+  mean the explicitly named lifetimes "come from nowhere."
+
+  Suggestion: Show off un-eliding the lifetimes in `get_erased` in this example.
+
+- [`BorrowedFd`](https://rust-lang.github.io/rfcs/3128-io-safety.html#ownedfd-and-borrowedfdfd)
+  uses these captured lifetimes to enforce the invariant that "if this file
+  descriptor exists, the OS file descriptor is still open" because a
+  `BorrowedFd`'s lifetime parameter demands that there exists another value in
+  your program that has the same lifetime as it, and this has been encoded by
+  the API designer to mean _that value is what keeps the access to the file
+  open_.
+
+  Its counterpart `OwnedFd` is instead a file descriptor that closes that file
+  on drop.
+
+## More to Explore
+
+- This way of encoding information in types is _exceptionally powerful_ when
+  combined with unsafe, as the ways one can manipulate lifetimes becomes almost
+  arbitrary. This is also dangerous, but when combined with tools like external,
+  mechanically-verified proofs _we can safely encode cyclic/self-referential
+  types while encoding lifetime & safety expectations in the relevant data
+  types._
+
+- The [GhostCell (2021)](https://plv.mpi-sws.org/rustbelt/ghostcell/) paper and
+  its [relevant implementation](https://gitlab.mpi-sws.org/FP/ghostcell) show
+  this kind of work off. While the borrow checker is restrictive, there are
+  still ways to use escape hatches and then _show that the ways you used those
+  escape hatches are consistent and safe._
+
+</details>