-
Notifications
You must be signed in to change notification settings - Fork 157
Memory Profiling
This page gives some steps for measuring how much memory we use with large number of virtual hosts in mod_pagespeed.
sudo apt-get install valgrind massif-visualizer
make apache_debug_install CONF=Release
(You want a release build since debug adds memory overhead)
make apache_install_conf
Open up ~/apache2/conf/pagespeed.conf in your favorite editor and remove the following:
ModPagespeedUsePerVHostStatistics on- The
<ModPagespeedIf>blocks
The debug config values for these settings contribute heavily to memory usage, and we will be recommending that large deployments don't use them.
Now, add in lots of vhosts to the config:
NUM_VHOSTS=8000 devel/lots_of_vhosts.sh --config >> ~/apache2/conf/pagespeed.conf
(Where NUM_VHOSTS= determines how many hosts are in use. The default is 10,000. Do not start with a really high value, or you will likely have to reach for the reset button).
Stop old apache:
make apache_debug_stop
Run Apache using valgrind's massif tool:
valgrind --tool=massif --threshold=0.1 --peak-inaccuracy=0.1 ~/apache2/bin/httpd
This may take some time to fully load up.
Unfortunately, you may also have to do this (and other parts of the experiment) twice: the above run will not include memory mmap'd by shared memory data structures, which can be quite significant. If you include --pages-as-heap=yes into the valgrind command it will be counted, but then for some heap allocations you will merely see when backing pages got allocated, rather than individual allocations that filled them.
If you want to see what the memory use looks after some traffic (and not just after bootup) some of the scripts can also help. You can use lots_of_vhosts.sh to generate a URL script that touches every host:
NUM_VHOSTS=8000 devel/lots_of_vhosts.sh --traffic > /tmp/traffic.txt
then devel/trace_stress_test.sh to throw it at the server:
PAR=2 devel/trace_stress_test.sh /tmp/traffic.txt
(the low PAR= is to reduce the risk of killing your machine and to spread memory load over fewer tasks)
First, stop apache:
make apache_debug_stop
This will also take a while, but after it you should end up with a whole bunch of massif.out.pid files. Pick one that looks typical, and open it in massif-vissualizer. That will give you a timeline of memory usage with a graph on the left and a tree on the right; if you click on the peak snapshot you can then expand the tree out to see where the allocations are coming from.