“Vlad and analysis of dtrace was used”

(title from the Google translation of a Japanese blog [edit: it is a technology news site, not a blog] about Firefox memory fragmentation)

Using dtrace and some tools that we’ve built I’ve been able to get more fragmentation data. I haven’t hooked up all the allocators yet — Vlad just made some changes that will make getting data from a bunch of the other allocators much easier, so expect more data soon.

Lets compare the Windows standard heap, the Windows Low-Fragmentation Heap, nedmalloc and jemalloc. I’ve posted pictures from the two Windows allocators before, but here they are again:

Windows standard heap — Small and compact, but very fragmented:

Windows Standard Heap

The Windows Low-Fragmentation Heap — bigger and less fragmented:

Windows Low-Fragmentation Heap

nedmalloc — faster but more fragmented than the Windows LFH:

nedmalloc

jemalloc — faster, smaller than Windows LFH, and less fragmented:

jemalloc

So far, I’m seeing that tcmalloc (the latest version, which I’m told is slower than previous releases) and nedmalloc are both about 10% faster at pure allocations than the Windows heaps (which are about the same speed). jemalloc looks to be also about 10% faster but I ran my tests on a different machine and need to verify my numbers before making a strong claim about its speed.

jemalloc looks to be a pretty solid contender. Jason Evans, jemalloc’s author has been super helpful in answering lots of questions I’ve had and has done some investigating of his own. I’ll hold off declaring a winner until I’ve had time to run with a few more allocators, but the data is showing that we can get good wins by switching allocators. I’m also in the process of generating different sets of logs to run against the allocators so we can see how they behave with stress tests such as loading 200 tabs and then closing them all.

On a side but related note, Olli Pettay and Jonas Sicking are doing lots of great work on moving content nodes and related data in to arenas.

15 thoughts on ““Vlad and analysis of dtrace was used”

  1. Gen Kanai

    Heh, actually that Japanese website is a technology news site, akin to a CNet or whatnot. What’s great is that they are bringing news about our work on fragmentation to Japanese readers.

    Reply
  2. fredrik

    Off-topic: what’s the deal with Planet.M.O randomly picking a word from the post and using that as the displayed title? For this entry on PMO the title was “jemalloc”. It’s been equally odd with earlier posts as well, but the feed looks fine. Weird.

    Reply
  3. Ian McKellar

    I wonder if all this work to make XPCOM objects GCable will allow us to move from pointers to a handle system so we can move them around. That would help fragmentation a *lot*.

    Reply
  4. jimis

    I think that the slowest but most space efficient settings, set by mallopt() would be:
    M_MXFAST 0 (disables fastbins for small allocations)
    M_TRIM_THRESHOLD 0
    M_MMAP_THRESHOLD 4*1024 (uses mmap() instead of brk() for all memory allocations?)
    M_MMAP_MAX 32*1024*1024 (and uses it even for large allocations, even if mmap() is slower than brk())

    Reply
  5. Pingback: More allocator data — tcmalloc edition « pavlov.net

  6. pavlov Post author

    Ian: everything but jemalloc is mostly cross-platform currently (aside from the platform allocators, obviously). I ported jemalloc to Linux and Mac in a few hours and Windows shouldn’t take much longer.

    Reply
  7. mark

    Would be curious to get your port of jemalloc to Linux. Would like to try with an application that is having memory fragmentation issues.

    Reply
  8. Pingback: Firefox 3 Memory Usage « pavlov.net

  9. brian v

    all this great effort, and still the installer is broken on server 2003. with b3, i could just keep running it a dozen times to get everything (xul.dll, etc). Now, half the dir.

    : – (

    Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s