flamenco, gossip: two-tiered value table#4800
Merged
ravyu-jump merged 2 commits intomainfrom May 2, 2025
Merged
Conversation
ravyu-jump
force-pushed
the
gossip-two-tiered-value-table
branch
3 times, most recently
from
8a3eff2 to
aa09a24
Compare
Contributor
cali-jumptrading
left a comment
cali-jumptrading
left a comment
There was a problem hiding this comment.
Not super familiar with the gossip implementation, so I did a quick pass. Looks great though!
ravyu-jump
force-pushed
the
gossip-two-tiered-value-table
branch
from
aa09a24 to
62e9553
Compare
ripatel-fd
reviewed
Apr 22, 2025
| vector as a push queue. | ||
| See fd_gossip_cleanup_values for an example */ | ||
| static ulong | ||
| fd_gossip_compact_values( fd_gossip_t * glob ) { |
Contributor
There was a problem hiding this comment.
Why preserve ordering in this algorithm? Unordered deletes have lower runtime complexity
Contributor
Author
There was a problem hiding this comment.
Yeah we technically don't need strict ordering, just need a way to make sure values that have been pushed and values that need to be pushed don't get mixed up. I can come up with something lower runtime complexity but that's probably better served in a separate PR
topointon-jump
previously approved these changes
Apr 28, 2025
…nd pullresp loops
convenience functions
ravyu-jump
force-pushed
the
gossip-two-tiered-value-table
branch
from
62e9553 to
3d31432
Compare
topointon-jump
approved these changes
May 2, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We now have two data structures that manage our Gossip value table
value_metas: holds the metadata (hash + wallclock) of a processed value. Used in dedup + bloom filter construction.
values: vector that holds the encoded form of the processed value. Used in gossip push + pull resp loops. The loops are now linear scans.
A cleanup policy that exploits this tiered value table will allow us to maintain a much smaller values vector without affecting the metadata map, which will help where only data is needed (push + pullresp). Will come in a future PR.