flamenco, gossip: two-tiered value table #4800
Conversation
ravyu-jump
force-pushed
the
gossip-two-tiered-value-table
branch
2 times, most recently
from
b273004 to
8a3eff2
Compare
…nd pullresp loops
ravyu-jump
force-pushed
the
gossip-two-tiered-value-table
branch
from
8a3eff2 to
aa09a24
Compare
convenience functions
ravyu-jump
force-pushed
the
gossip-two-tiered-value-table
branch
from
aa09a24 to
62e9553
Compare
| vector as a push queue. | ||
| See fd_gossip_cleanup_values for an example */ | ||
| static ulong | ||
| fd_gossip_compact_values( fd_gossip_t * glob ) { |
There was a problem hiding this comment.
Why preserve ordering in this algorithm? Unordered deletes have lower runtime complexity
There was a problem hiding this comment.
Yeah we technically don't need strict ordering, just need a way to make sure values that have been pushed and values that need to be pushed don't get mixed up. I can come up with something lower runtime complexity but that's probably better served in a separate PR
| @@ -376,6 +376,8 @@ | |||
| | gossip_​peer_​counts_​repair | `gauge` | Number of peers of each type (Repair) | | |||
| | gossip_​peer_​counts_​voter | `gauge` | Number of peers of each type (Voter) | | |||
| | gossip_​shred_​version_​zero | `counter` | Shred version zero | | |||
| | gossip_​value_​map_​size | `gauge` | Current size of the CRDS value map | | |||
There was a problem hiding this comment.
Could we make it clear this is the value_metas
| static void | ||
| fd_gossip_cleanup_values( fd_gossip_t * glob, | ||
| fd_pending_event_arg_t * arg ) { | ||
| (void)arg; |
There was a problem hiding this comment.
nit: FD_PARAM_UNUSED
| if ( ele->wallclock<value_expire ) { | ||
| /* This value has expired, mark it for deletion in the value vector and remove from map */ | ||
| if( ele->value!=NULL ){ | ||
| ele->value->del = 1UL; |
There was a problem hiding this comment.
Do you mind elaborating in a comment why we need the two-stage deletion?
| memmove( &vec[start - num_deleted], | ||
| &vec[start + 1], | ||
| (next - start - 1) * sizeof(fd_value_t) ); |
There was a problem hiding this comment.
could we document that the motivation for this is to reduce memory footprint
We now have two data structures that manage our Gossip value table
value_metas: holds the metadata (hash + wallclock) of a processed value. Used in dedup + bloom filter construction.
values: vector that holds the encoded form of the processed value. Used in gossip push + pull resp loops. The loops are now linear scans.
A cleanup policy that exploits this tiered value table will allow us to maintain a much smaller values vector without affecting the metadata map, which will help where only data is needed (push + pullresp). Will come in a future PR.