database: impl service::{Request,Response} mappings

[hinto-janai]

Part 19

#98 <- Previous Next -> #102

• Updates to cuprate_types::service::{Request,Response} types
• Adds fn signatures for the service reader/writer threads (impl after database: implement ops/ #102 is merged)
• Implements service reader/writer thread request/response/fn mapping logic
• Makes Env::resize_map() return the new memory map size

[hinto-janai]

@Boog900 As add_block() will take VerifiedBlockInformation by value so it can just take the inner data instead of cloning, the resize retry problem here is fixed using #101 (comment).

Another option is just setting the memory map to a large value at startup (e.g. 16TB) such that this error never occurs - I'm not sure what the overhead in doing this is though.

[Boog900]

@hinto-janai Why does add block need to take by value? it should be able to just reference the data

[Boog900]

I looked into setting the memory map size high and it supposedly works on linux but I found a few people mentioning problems on windows/ other platforms, maybe these have been fixed but I would rather not do this if we don't have to.

[hinto-janai]

Why does add block need to take by value? it should be able to just reference the data

Mostly because block blobs is StorableVec, which is just StorableVec(Vec<T>) so we can't pass a &Vec<T> here: https://github.com/hinto-janai/cuprate/blob/1a11938701df3be9165e45078b94f0d44b212d22/database/src/ops/block.rs#L90-L93

I guess we could clone it inside add_block() but that means callers don't get to decide when to clone, i.e. even if they don't need the VerifiedBlockInformation after add_block() it'll still get cloned.

[Boog900]

Mostly because block blobs is StorableVec, which is just StorableVec(Vec)

you can ref cast: https://docs.rs/bytemuck/latest/src/bytemuck/transparent.rs.html#142-155

[hinto-janai]

Right... I forgot, will make add_block() take by ref.

I'll revert it so RuntimeError::ResizeNeeded is handled reactively too. Does it make sense to retry writes a 3rd time? If the 2nd try did not resize correctly something is surely wrong and we should panic, right?

[Boog900]

I don't think there is anything wrong with trying 3 times though, better to go to high than to low IMO

[hinto-janai]

Sorry, should be ready now.