In the vein of what has been done for openoffice, the imminent release of Firefox 3 gave us the idea to try to test the performance improvements in the version 3 compared to the various Firefox browser versions. Since it’s our preferred platform, all our tests were made on Linux (see platform for a precise description of the benchmark platform).
Looking at the graph:
- interestingly, lots of the performance improvements seems to have taken place during the beta cycle
Memory usage: worse than Firefox 2?
In his blog post, Firefox developer Stuart Parmenter presents a benchmark on the improvements made in Firefox 3 memory usage. Since we have always been amazed on how poorly Firefox performs on Linux compared to Windows, we decided to use quite the same testing process on Linux. For the sake of completeness, we also added other Firefox versions into the benchmark.
Site Map The “Stuart Parmenter’s membuster test” consist of the loading of a bunch of 29 different web pages through 30 windows over 11 cycles (319 total page loads), always opening a new window for each page load. Every browser is configured to use an aggressively caching proxy (Squid) to minimize network latency effects on the measurement (although the memory measurement should not depend on that). The test lasts 10 minutes and tries to mimic a normal browsing behavior. We measured the RSS memory of Firefox every 5 seconds (see this post for an explanation of the tricky subject of Linux memory measurement … and to get a good understanding of the limitations of our choice to measure the RSS memory).
Of course RSS measurement is not really accurate in terms of absolute memory measurement but since what matters here is only relative memory measurement (cf. memory usage reloaded for a better experiment) we thought it might be significant.
The results are quite astonishing:
- Firefox version 3 seems to eat more memory than the version 2 (quite the opposite as what Stuart Parmenter found on Windows!)
- the increase in memory over time seems to be slightly worse for Firefox 3
Following Vladimir Vukicevic’s advice we try to overcome RAM measurement weaknesses by using a more accurate estimation of the RAM used (directly and indirectly) by Firefox; we also measured the RAM memory used by the Xorg process. Please note that those RAM measurements are not interesting in absolute value but more compared to each other.
All the tests were made on a freshly booted machine with as less running applications as possible. We know that RAM measurement is really tricky and never perfect but we tried to do our best. We also made a longer test than the previous one and close all windows except one around 17’ to see how Firefox releases memory.
- the new way of measuring the memory does not lead to significant changes compared to RSS measurement (cf. dotted lines)
- if you consider the Xorg + Firefox RAM usage (solid lines), the difference is less significant (probably due to new ways of handling the memory in FF3) but Firefox 2 still performs slightly better than in version 3
- one point where the efforts of Firefox developers seems to have paid off is the release of memory during idle time: the dropoff at 17’ when browsing stops is bigger with Firefox 3
All measurements have been made on an Asus F2J-5E020P with 2Go RAM (we thought this should suffice to avoid swapping) running Kubuntu Gutsy Gibbon.
Although the improvement in how Firefox releases memory once the browsing has stopped is significant, after so much advertisement over the improved memory usage, the tests we made on a Linux platform were quite disappointing in terms of memory usage during browsing (compared to the results on windows) and we must conclude that most of the work done on memory management will mostly benefit Windows users.
[update 1/12:00 June 17th GMT] following
[update 2/12:00 June 18th GMT] after a discussion with Firefox developers, we tested the RAM usage again to measure how Firefox releases memory after any browsing has stopped and updated our graphs