optimized extranonce prefix generation

[coleFD]

PR #1924

EDIT: This is just a proof of concept to show a better way to manage extranonce prefixes. Not looking to get this merged. I am preoccupied with other things atm

[coleFD]

id is hardcoded to 4 bytes because it gives 2 bytes (65,536 connections) which is more than enough, and another 2 bytes for the server_id. This could be changed to support 3 or 4 as well but not sure if it's worth the extra byte. I dont see a reason to support more than 4 bytes

[plebhash]

it gives 2 bytes (65,536 connections)

are we assuming that each connection will only consume one single extranonce_prefix? what if the same connection has multiple channels?

also, what's the rationale behind the number 65,536?

[coleFD]

rather than connections, i should say channels or "unique ids".

65,536 is a u16::max (2 bytes). It's fairly easy to make this size configurable to whatever size the user wants with generics.

[coleFD]

But an added benefit of restricting the total extranonce-prefix to 4 bytes is that it could be used as a channel id as well

[GitGab19]

@coleFD what do you think about using 1 byte for the server_id (which would support 256 different servers), and 3 bytes for the extranonce_prefix(es) (which would support up to 16.777.216 different channels)?

[coleFD]

that works too. You could allow this to be "configurable" with generics. Or maybe the optimal is somewhere in between like 12 bits for server ID (4096 servers) and 20 bits (1.05M) for the channel component. I think it's just preference. One thing to note is that as the channel component increases, finding a free bit gets slower since it's O(n). at 2 bytes, i benchmarked somewhere around 1-2 micros for finding available bits which is fine considering this is only called once per channel at origination. There may be a better way to start at the last freed bit index as opposed to .skip_while() though. Either way, performance should be considered because it is not uncommon to drop/connect a bunch of miners as fast as possible (ie. curtailment and non-agg proxies) and the connection time across thousands of clients add up.

[coleFD]

Another thing to note which may point out the benefit to extra serverID bits is it can be used to distinguish between clusters in different regions, like 1 byte serverID + 1 byte cluster ID + 2 bytes channel component. In this case the api can recognize this as 2 bytes server ID + 2 bytes channel component where it is up to the user to ensure the server ID is unique with the cluster ID. This is the route we are going

[jbesraa]

The code is exposing two structs with the same name and now an ExtranoncePrefix/Manager is added that is NOT using any of them. I think this is too confusing. If the intention is to keep the three of them(two of them is already confusing enough IMO) - a good documentation explaining when to use what is crucial.

stratum_apps::stratum_core::mining_sv2::Extranonce
stratum_apps::stratum_core::sv1_api::utils::Extranonce

[coleFD]

The code is exposing two structs with the same name and now an ExtranoncePrefix/Manager is added that is NOT using any of them. I think this is too confusing. If the intention is to keep the three of them(two of them is already confusing enough IMO) - a good documentation explaining when to use what is crucial.

stratum_apps::stratum_core::mining_sv2::Extranonce
stratum_apps::stratum_core::sv1_api::utils::Extranonce

This is just a proof of concept MR to show how extranonce prefix can be safely compacted to save bytes in the coinbase script. It isn't meant to be merged, at least not as is.

[GitGab19]

This approach based on bitvec is quite interesting, and way better than the simple counter we currently have.

Instead of defining it as a new additional struct and module, though, I would unify it with the one we have in https://github.com/stratum-mining/stratum/blob/main/sv2/subprotocols/mining/src/lib.rs.

In the current module we have some interesting pieces which I would keep, such as the different ranges (which could be renamed into something more descriptive) to represent the ExtendedExtranonce.

I would also keep some methods we have there, which are useful for JDC and Translator, such as: from_upstream_extranonce, next_prefix_standard, next_prefix_extended.

We could nuke the old module there, and merge the aforementioned old pieces with this bitvec idea under a new dedicated module for extranonce in channels_sv2 (as in your prototype here).