mininglabs

Data mining, analysis and visualization

Firefox 3: an empirical performance study

In the vein of what has been done for openoffice, the imminent release of Firefox 3 gave us the idea to try to test the performance improvements in the version 3 compared to the various Firefox browser versions. Since it’s our preferred platform, all our tests were made on Linux (see platform for a precise description of the benchmark platform).

Our method allows us to present some numbers on how well the much awaited Firefox 3 release is doing in terms of javascript performance and memory usage on Linux.

Javascript performance: a clear improvement

We used the well known Sunspider Javascript 0.9 test platform for different versions of Firefox:

Looking at the graph:

• Firefox 3.0 features a dramatic performance improvement on the javascript engine
• interestingly, lots of the performance improvements seems to have taken place during the beta cycle

Memory usage: worse than Firefox 2?

In his blog post, Firefox developer Stuart Parmenter presents a benchmark on the improvements made in Firefox 3 memory usage. Since we have always been amazed on how poorly Firefox performs on Linux compared to Windows, we decided to use quite the same testing process on Linux. For the sake of completeness, we also added other Firefox versions into the benchmark.

The “Stuart Parmenter’s membuster test” consist of the loading of a bunch of 29 different web pages through 30 windows over 11 cycles (319 total page loads), always opening a new window for each page load. Every browser is configured to use an aggressively caching proxy (Squid) to minimize network latency effects on the measurement (although the memory measurement should not depend on that). The test lasts 10 minutes and tries to mimic a normal browsing behavior. We measured the RSS memory of Firefox every 5 seconds (see this post for an explanation of the tricky subject of Linux memory measurement … and to get a good understanding of the limitations of our choice to measure the RSS memory).

Of course RSS measurement is not really accurate in terms of absolute memory measurement but since what matters here is only relative memory measurement (cf. memory usage reloaded for a better experiment) we thought it might be significant.

The results are quite astonishing:

• Firefox version 3 seems to eat more memory than the version 2 (quite the opposite as what Stuart Parmenter found on Windows!)
• the increase in memory over time seems to be slightly worse for Firefox 3

Memory usage reloaded

Following Vladimir Vukicevic’s advice we try to overcome RAM measurement weaknesses by using a more accurate estimation of the RAM used (directly and indirectly) by Firefox; we also measured the RAM memory used by the Xorg process. Please note that those RAM measurements are not interesting in absolute value but more compared to each other.

All the tests were made on a freshly booted machine with as less running applications as possible. We know that RAM measurement is really tricky and never perfect but we tried to do our best. We also made a longer test than the previous one and close all windows except one around 17’ to see how Firefox releases memory.

• the new way of measuring the memory does not lead to significant changes compared to RSS measurement (cf. dotted lines)
• if you consider the Xorg + Firefox RAM usage (solid lines), the difference is less significant (probably due to new ways of handling the memory in FF3) but Firefox 2 still performs slightly better than in version 3
• one point where the efforts of Firefox developers seems to have paid off is the release of memory during idle time: the dropoff at 17’ when browsing stops is bigger with Firefox 3

Platform

All measurements have been made on an Asus F2J-5E020P with 2Go RAM (we thought this should suffice to avoid swapping) running Kubuntu Gutsy Gibbon.

Conclusion

In conclusion, we must say that we were the first surprised by our results. The javascript improvement is quite impressive (and anybody who has the curiosity to test the Firefox 3 release candidates has had the “feeling” of the improved Firefox speed with javascript heavy pages).
Although the improvement in how Firefox releases memory once the browsing has stopped is significant, after so much advertisement over the improved memory usage, the tests we made on a Linux platform were quite disappointing in terms of memory usage during browsing (compared to the results on windows) and we must conclude that most of the work done on memory management will mostly benefit Windows users.

[update 1/12:00 June 17th GMT] following Vladimir Vukicevic’s advice we added the “memory usage reloaded” section presenting more accurate memory measurement and updated the conclusion accordingly

[update 2/12:00 June 18th GMT] after a discussion with Firefox developers, we tested the RAM usage again to measure how Firefox releases memory after any browsing has stopped and updated our graphs