[GEN][ZH] Refactor the recorder class to use a ramfile for replay playback.

[Mauller]

This PR converts the Recorder class to use the game file system for replay loading and playback.

This was required to allow the use of a RAMFile which allows the replay to be fully loaded into ram while being accessed through the common file handling interface.

This will need some testing in headless playback, i saw a small improvement in replay playback times under normal circumstances. But the biggest improvements will likely be under headless mode.

When other bottlenecks are removed in future, this change should show bigger contributions to replay playback speed.

[Mauller]

@helmutbuhler @roossienb Would be worth giving this a test under headless to see what impact it has overall.

[helmutbuhler]

@helmutbuhler @roossienb Would be worth giving this a test under headless to see what impact it has overall.

I did 4 runs with several replays in sequential order for each run. 2 runs without this PR, 2 runs with this PR:
Total time of all replays in each run without this PR: 0:08:51 and 0:09:04
With this PR: 0:08:44 and 0:08:49

The times in more details are here:
ramtest.zip

So is this a significant improvement or just random noise? I don't know...

But I should also mention that I don't see any significant diskaccess when simulating replays, not even when doing 24 replays in parallel.

[Mauller]

That is fair, we should see more improvements coming from this change when other bottlenecks during playback are removed.

There won't be significant amounts of disk access overall, it's just the fact it's a lot of small reads that happen. Which most drives don't like, but overall reading from disk is always a bottleneck compared to RAM.

This refactor also makes the playback file handling code easier to understand as well. The writing code is another matter entirely but i left that in place to use the original m_file and it is not that big an issue to convert it over to use the game file system.

[Mauller]

This should be good to go if we wanted to consider using this change.

[xezon]

I was hoping for more performance gains but what helmut measured does not look exciting. We can still move forward with this change because it is technically an improvement.

[xezon]

This means that replay playback loses the ability for file streaming, aka reading a replay while it is being written to. Can we make this configurable in a way?

[Mauller]

is it common to do this? i could look at adding an option in the options.ini to allow it to disable using the ram file if set, otherwise use the ram file by default.

[xezon]

As far as I am aware Legionnaire uses this. Note that the LocalFile has a performance issue:

• Raw file reads and writes are unbuffered #1074

I am currently looking into that.

[Mauller]

updated with a rename to the original m_file to call it m_recordingFile to differentiate it from the replay playback file.

[helmutbuhler]

I was hoping for more performance gains but what helmut measured does not look exciting. We can still move forward with this change because it is technically an improvement.

@roossienb mentioned that he has heavy diskaccess during simulation. Maybe he can test whether this PR is an improvement.

[roossienb]

I was hoping for more performance gains but what helmut measured does not look exciting. We can still move forward with this change because it is technically an improvement.

@roossienb mentioned that he has heavy diskaccess during simulation. Maybe he can test whether this PR is an improvement.

Tried to do some with running -jobs 100 for 300 replays and pushing it to the limit. Got 512 seconds, 1019 seconds and 842 seconds. Way too much noise and other factors to get a proper performance analysis.

[helmutbuhler]

I was hoping for more performance gains but what helmut measured does not look exciting. We can still move forward with this change because it is technically an improvement.

@roossienb mentioned that he has heavy diskaccess during simulation. Maybe he can test whether this PR is an improvement.

Tried to do some with running -jobs 100 for 300 replays and pushing it to the limit. Got 512 seconds, 1019 seconds and 842 seconds. Way too much noise and other factors to get a proper performance analysis.

Unless you have some monster pc, -jobs 100 will just run the scheduler crazy ;) I suggest put the number after -jobs as many cores you have, and then watch in taskmanager whether disk utilization is better when this pr is applied. Or run without jobs and measure the time.