Holy RAM, batman! (Out of memory errors and excessively high memory usage)

[bclothier]

After observing random "System resources exceeded" or "Out of memory" with my usercode which didn't quite make sense, I later ran quick fixes. After applying changes, I went to start the application and immediately got the System resources exceeded *despite have NOT run any VBA code`.

Suspicious, I restarted Access, then opened Access. Noted the working set memory at 8MB.
Launch open VBA, the working set rockets to 175MB working set memory.
Run parse, the working set goes up to 300 MB!
Open quick inspector, and refresh, the working set climbs to 475 MB!

I might be going out on a limb but it sure looks like to me something's hemorrhaging hemorrhages.

[retailcoder]

Smells like a memory leak indeed. However note that much of the performance comes from caching - trade-off is between memory consumption and performance. Not saying there's no room to consume less memory for the same performance, but RD isn't, and never was, intended to be lightweight at all.

[bclothier]

@retailcoder I'm totally down with RD never ever being lightweight. I wouldn't expect it, really.

However, I think it is worthwhile to point out that I can load 2,282,482 records in an Access datasheet and only have 25-30 MB in the working set. I can do other operations such as grouping much faster. You can try it out yourself by playing w/ FactSale table from this package: http://powerpivotsdr.codeplex.com/releases/view/46355 Note: technically, that's a lie - Access merely makes it looks like it loaded that many but it in fact maintains a scrolling window, having ~100 records filling in background and partially in response to user's action; it only needs to load 2 million keys, then transverse the chain to read the whole data.

I know this is very an apple to oranges comparison because I'm comparing a C++ application against C# application but I also point out that Access is fundamentally disk-based which is an order of magnitude slower than RAM and we still see good performance nonetheless.

I am not well-qualified to offer criticism but it does not appear to me we are using database but primarily in-memory collections. That makes me think many operations we are doing are always going to be expensive because it isn't a database and has no index to reduce the search space. It might be possible that RD would benefit more from using sqlite or something similar than caching service that is entirely in memory, given the large number of objects RD needs to have and comparatively only a small subset get invalidated after some event. Having tables and indexes would allow you to quickly prune and re-populate the subsets, likely in spite of saving to a much much slower subsystem.

Just a thought.

[retailcoder]

it isn't a database and has no index to reduce the search space.

Most of our lookups are hash matches in dictionary keys, an essentially O(1) operation that completes pretty much instantaneously regardless of how many thousands of keys we're looking at... And it's all in RAM indeed. An empty project with standard library references is working with ~50-60K declarations, and we cache a number of aggregates too. Each declaration is an object that contains information about its name, type, accessibility, references in user code, etc.; this metadata is fundamental for Rubberduck to understand the code.

To be honest it never even occurred to me that we could use a database and literally query the code. That's a very, very interesting idea...

[bclothier]

In that case, if you do use a database, your ideal database should support hash indexes as to keep the O(1) operation. I can't recall off the cuff if sqlite has hash index. I wrongly(?) assumed that you were also doing querying because in the quick fix inspection window, we group stuff, and display only first 100. That's an example of querying where a B-tree index can help cut down on the search space because you don't want to literally loop each single member of the collection in this case. You only want a scrolling window of data.

From reading I did whenever you invalidate a module reference, you have to delete all references from that module, then re-build it. That's another example where a B-tree index would win out on a hash index -- just delete all references with module = "x", done in a sequential scan. Besides, it's much easier to optimize a database than it is code (at least that's the case for me).

[retailcoder]

Oh, I was merely talking about the DeclarationFinder, which is where most queries begin; inspections use it to get all user declarations, or all byref parameters, or line labels, or variables, whatever the inspections inspect. Using a database backend would be a complete gsme changer... I'm giving this some serious thought.

[Vogel612]

FWIW @bclothier sqlite does not maintain a hash-index. It does maintain some indexes, but it's IMO not suited to replace the mass of frequently updated information that Rubberduck needs. SQLite is simply not intended for the load of more or less completely refreshing a whole database whenever RD reparses.

[bclothier]

Just to give an apple-pear comparison... I fired up VIsual Studio 2015 and loaded Rubberduck solution (what else?).

Initial loading took about ~245 MB of working set.

Building it bumps it up to ~280 MB.

Running code analysis on it adds some more up to ~315-320 MB.

This is keeping in mind that Visual Studio is much more complicated and it likely has to handle far more references & declaration to support code analysis (which might not be as sophisticated as Resharper). That said, it's safe to say it doesn't need to increase by more than 100MB just to parse or to do inspection.

Whether Visual Studio is doing this all in-memory or not, I don't know.

That said, I question whether you really need to "completely refreshing a whole database whenever RD parses." Given that I can only type so fast and that I can only change one module at a time (which remains true even if I'm using VBIDE to automate my changes), you're always going to be invalidating a subset. Even if I clicked Parse, it'd be still more logical to parse what changed unless I explicitly force a full parse (e.g. akin to Rebuild method in Visual Studio). So tossing the whole database should be, IMO, rare and explicit.

And if you are always going to work in subsets, you aren't going to get good performance from in-memory collections that can only support hash indexing; you need a range seek so you can enumerate only that subset, done.

Maybe your answer is a in-memory collection that supports B-tree indices. Maybe it's really just a boring old memory leak.I don't know. But I really don't think we need that much RAM based on what I've observed.

[retailcoder]

Visual Studio is also at version 15 or so (we work with v6.0 essentially), with Roslyn being the single most efficient compiler/analyzer out there, written by an army of brains. We are not Microsoft, or JetBrains, and we're not hundreds of contributors, we're a handful, doing this part-time when it's possible. It is thoroughly unfair a comparison. VS & Roslyn are integrated, they can parse a single modified line of code. Rubberduck does what it can with the shitty VBIDE API, and parses with ANTLR, and because it's not integrated the smallest possible granular unit we can parse is an entire module - anything else and we can kiss good-bye token positions. So yes, a subset - but not half as granular as a single modified line of code.

That said I don't think we would have to ditch & recreate the whole database every time. A db could be a very interesting solution to project metadata, per-project and per-user settings, and for storage of the thousands of COM declarations, which definitely make up the bulk of the RAM we're eating up.

[bclothier]

@retailcoder I apologize if I've overstepped. You're absolutely right that VS has much more resources behind it than RD does, so what RD people achieved is nothing short of phenomenal. I clearly didn't think about ANTLR vs Roslyn. I did want to get a sense of what is possible, hence my comparison. That said, I'll stop putting on big britches now.

[retailcoder]

@bclothier I didn't mean to come off rude or even annoyed - at the end of the day RD is eating up a ton of RAM, and likely leaking some; memory leaks are a critical issue, and embedding an actual database isn't a completely crazy idea at all. I think I'd go with SQL Server Express though. Gotta think about how to deploy RD to the website too (that online parser in the inspections page is performing terribly ATM).

Thanks for your ideas! 😄

[retailcoder]

[Vogel612]

#3405 could be related? We should reevaluate this after a fix for that is merged