LMDB write throughput limitations when rescanning large numbers of accounts

[Shadowfoxd]

I’ve been testing Monero LWS back-end and noticed some performance bottlenecks related to LMDB’s write limitations.

Observed Behaviour

LWS workloads involve frequent blockchain rescans and wallet updates across thousands of accounts.

Since LMDB only supports a single writer transaction at a time, high-volume re-scanning causes writes to queue up and slow down significantly.

With ~10,000+ accounts, re-scanning throughput drops drastically compared to expected performance.

Questions

Are there recommended best practices for tuning LWS + LMDB for frequent writes (like re-scanning thousands of accounts)?

Would introducing write batching, delayed commit strategies, or alternative back-ends (e.g., RocksDB/SQLite or any alternatives ) help in practice?

Has there been any bench marking or consideration of LMDB alternatives for high account-count workloads?

Thank in advance

[vtnerd]

Since LMDB only supports a single writer transaction at a time, high-volume re-scanning causes writes to queue up and slow down significantly.

The two bottlenecks are: (1) each thread does a unique ZMQ call to the daemon making the bottleneck inside monerod itself, and (2) LMDB account writes are blocking on local scan threads. You've identified (2) as the issue, and I'm assuming you've done some profiling to determine this issue.

The current implementation has a blocking write on each local scan thread, whereas the remote scan feature uses asynchronous writing. A quick test on your 10,000 account setup would be to run a remote scan locally, i.e. run --scan-threads 0 on monero-lws-daemon and run monero-lws-client on the same box. In this setup, each scan thread in monero-lws-client forwards all account writes to monero-lws-daemon for writing on a single I/O thread. This is primarily testing whether it's faster to convert an account to msgpack than wait for a LMDB write.

There's a decent chance this is faster, even with TCP overhead. If the numbers are better (or even close), then the next step is converting the scan threads local to monero-lws-daemon to asynchronous setup too. This will be even faster because we can "clone" (copy) the account without converting to msgpack and can use a high throughput queue instead of TCP.

Lastly, converting the local scan threads to asynchronous should be somewhat simple, so even if the TCP test is mediocre, this probably worth investigating.

With ~10,000+ accounts, re-scanning throughput drops drastically compared to expected performance.

Do you have any hard numbers for your setup?

Are there recommended best practices for tuning LWS + LMDB for frequent writes (like re-scanning thousands of accounts)?

Not yet. Trying the remote scan trick is the only one I can think of right now.

Would introducing write batching, delayed commit strategies, or alternative back-ends (e.g., RocksDB/SQLite or any alternatives ) help in practice?

Adding an alternative backend is likely difficult. RocksDB is probably the best fit if someone wanted to attempt it. Most of the db operations are contained in src/db/storage.cpp but everything is hand-tuned for LMDB.

The long-term solution is probably making all account writes asynchronous. The I/O thread should be able to "keep-up", but it may depend on the storage device.

[vtnerd]

There's a decent chance this is faster, even with TCP overhead. If the numbers are better (or even close), then the next step is converting the scan threads local to monero-lws-daemon to asynchronous setup too. This will be even faster because we can "clone" (copy) the account without converting to msgpack and can use a high throughput queue instead of TCP.

Actually due to design, it may end up using msgpack anyway, as this I/O thread is an ASIO loop (for better or worse).