Reading entire file in memory

[abozhilov]

From what I can see and from my tests, the entire mmdb file is loaded in the memory.
Would be great if the lookup tree is traversed without loading the whole tree in memory as mmdb format is optimised for traversals like that.

[runk]

Memory is cheap these days, and most cases require high performance for address lookup. With random fs reads it's nowhere close to desired performance.

I used hybrid approach in the beginning where only index is in memory, and data section read from filesystem. Later it changed for performance reasons. That said, you're welcome to make a contribution to introduce pluggable "backends" which can be configured to be fs only, leaving the full in-memory mode as default.

[abozhilov]

I have deleted my previous comment, because I thought it's my mistake, but actually it's not.
This library unfortunately eats a lot of memory, and seems it's proportionally to the number of lookups. I have a process with a lot of lookups and 2GB of memory is not enough.
Yeah, memory is cheap these days, but when you have thousands of processes which being charged for the consumed memory you may think otherwise.

[runk]

4x of the db size (512mb) sounds a bit excessive indeed.

• Are you sure it's all attributed to maxmind lib?
• Are you sure you initialise it once only?
• What's your cache.max setting?

If you use lambda, its performance is generally controlled by the allocated memory. More memory it has, faster it runs (unless it does slow I/O over network) - this is the case even if not all memory is utilised. So paying for 2 times less memory often means your fn runs 2x slower, which generally means paying same money for slower running functions.

I'd appreciate a snippet which proves that 4x memory usage is solely attributed to the maxmind lib, maybe something similar to #886