Firefox 3: an empirical performance study

In the vein of what has been done for openoffice, the imminent release of Firefox 3 gave us the idea to try to test the performance improvements in the version 3 compared to the various Firefox browser versions. Since it’s our preferred platform, all our tests were made on Linux (see platform for a precise description of the benchmark platform).

Our method allows us to present some numbers on how well the much awaited Firefox 3 release is doing in terms of javascript performance and memory usage on Linux.

Javascript performance: a clear improvement

We used the well known Sunspider Javascript 0.9 test platform for different versions of Firefox:

Figure #1: Firefox javascript performance (smaller is better)

Looking at the graph:

Memory usage: worse than Firefox 2?

In his blog post, Firefox developer Stuart Parmenter presents a benchmark on the improvements made in Firefox 3 memory usage. Since we have always been amazed on how poorly Firefox performs on Linux compared to Windows, we decided to use quite the same testing process on Linux. For the sake of completeness, we also added other Firefox versions into the benchmark.

Site Map The “Stuart Parmenter’s membuster test” consist of the loading of a bunch of 29 different web pages through 30 windows over 11 cycles (319 total page loads), always opening a new window for each page load. Every browser is configured to use an aggressively caching proxy (Squid) to minimize network latency effects on the measurement (although the memory measurement should not depend on that). The test lasts 10 minutes and tries to mimic a normal browsing behavior. We measured the RSS memory of Firefox every 5 seconds (see this post for an explanation of the tricky subject of Linux memory measurement … and to get a good understanding of the limitations of our choice to measure the RSS memory).

Of course RSS measurement is not really accurate in terms of absolute memory measurement but since what matters here is only relative memory measurement (cf. memory usage reloaded for a better experiment) we thought it might be significant.

Figure #2: Stuart Parmenter’s membuster test

The results are quite astonishing:

Memory usage reloaded

Following Vladimir Vukicevic’s advice we try to overcome RAM measurement weaknesses by using a more accurate estimation of the RAM used (directly and indirectly) by Firefox; we also measured the RAM memory used by the Xorg process. Please note that those RAM measurements are not interesting in absolute value but more compared to each other.

All the tests were made on a freshly booted machine with as less running applications as possible. We know that RAM measurement is really tricky and never perfect but we tried to do our best. We also made a longer test than the previous one and close all windows except one around 17’ to see how Firefox releases memory.

Figure #3: Stuart Parmenter’s membuster test (Firefox and Xorg) using ps_mem

Platform

All measurements have been made on an Asus F2J-5E020P with 2Go RAM (we thought this should suffice to avoid swapping) running Kubuntu Gutsy Gibbon.

Conclusion

In conclusion, we must say that we were the first surprised by our results. The javascript improvement is quite impressive (and anybody who has the curiosity to test the Firefox 3 release candidates has had the “feeling” of the improved Firefox speed with javascript heavy pages).
Although the improvement in how Firefox releases memory once the browsing has stopped is significant, after so much advertisement over the improved memory usage, the tests we made on a Linux platform were quite disappointing in terms of memory usage during browsing (compared to the results on windows) and we must conclude that most of the work done on memory management will mostly benefit Windows users.

[update 1/12:00 June 17th GMT] following Vladimir Vukicevic’s advice we added the “memory usage reloaded” section presenting more accurate memory measurement and updated the conclusion accordingly

[update 2/12:00 June 18th GMT] after a discussion with Firefox developers, we tested the RAM usage again to measure how Firefox releases memory after any browsing has stopped and updated our graphs

If you enjoyed this post, please consider to leave a comment or subscribe to the feed and get future articles delivered to your feed reader.

Comments

Well, looks like Mozilla guys had spend a lot of effort on Windows version and not so much on Linux, again. Let’s see what will they say about it with the FF3 release, and thank you very much for the data and graphics.

Thank you for validating my own personal experience with FF3 under Linux, and for providing these graphics.

Everyone thought I was crazy whenever I mentioned the fact that FF3 under Linux was a memory hog and markedly worse than FF2.

Yes, it seems the FF devs didn’t pay enough attention to the Linux version, but if I remember correctly the early versions of FF2 also suffered from the same neglect.

Firefox 3 has more code size than Firefox 2, which might account for some of the RSS difference. Have you tried the script at http://www.pixelbeat.org/scripts/ps_mem.py (referenced in a comment in the article you linked)? It seems to be better at reporting the actual amount of heap memory in use.

You may want to measure the memory usage of the X process as well, because I think you need to look at the additional usage there as well. Because of the way X works, many images end up being stored on the X server (which in turn might store them in video memory, but that’s rocket science to measure). On linux, we didn’t see as big wins from the new allocator as we did on Win32, since Linux’s allocator was already decent, but there were improvements. (We track memory usage on all three of our main platforms on an hourly basis through our automated build and test infrastructure, and any regression is noticed quickly.)

Hi Vladimir! Thanks a lot for your comment/advice. We were aware of the limitations of the RSS memory measurement but we thought it could be significant enough to just compare the different versions (we tried the ps_mem script without getting any real difference [see fig. #3 dotted lines]). We followed your advice and added a section to plot Xorg + Firefox RAM usage. The memory usage difference is much less important between v2 and v3 but still a little bit in favor of Firefox 2.

[…] http://www.mininglabs.com/2008…; Seems Mozilla spent more time developing for windows and NOT Linux, as the chart shows increased memory usage which confirms what I just saw.. […]

[…] ler dois posts na web sobre as fracas performances do novo Firefox3 em GNU/Linux e da minha própria experiência, resolvi fazer o download do Flock 2.0Beta1, é sem […]

I fear you must study how operating systems work before such claims.

UNIX like systems, like GNU/Linux, try to maintain in memory as much as possible (even after applications exit) in order to offer better performance.

From my experience, I notice significantly better speed and memory usage.

Our claim is that, except from the memory release during idle time, there is no significant improvement in memory management on Linux, especially compared to what the study on Windows shows.

We know that Unix memory strategy is to cache as much as possible but this does not mean that our test is invalid: what we do here is _compare_ how Firefox versions use memory during the same browsing session.

A limit of our approach (from a discussion with Firefox developers) is that there are still some applications interacting with Firefox (besides Xorg) that we should be monitoring in order to get a very precise picture of the memory used (like pango or fontconfig).

[…] información | Pavlov. Más información | MiningLabs. En Genbeta | Especial Firefox 3. trackback ¿Recomendarías este post? Más noticias […]

[…] información | Pavlov. Más información | MiningLabs. En Genbeta | Especial Firefox […]

Abbreviation of Firefox is Fx or fx, not FF nor ff.

Abbreviation of Firefox is Fx or fx, not FF nor ff.
and woe betideƒ thee who dare speaketh other than the king’ƒ tongue.
i wonder what’s the ram comparison on linux, for opera 9.5 vs ff3?

it might be an improvement otherwise, but in terms of memory usage Firefox is becoming bad in all their releases after V2.0 which Flock is using - so am not quite sute
I would have never wanted to upgrade and dont like this feature of firefox of compulsory upgradation - came to firefox as it was light and faster and now I believe its losing its edge, but surely has become popular and people like us are permanently switched to Firefox.

Leave a comment

(required)

(required)