September 27, 2011

Finding memory leaks in Python with objgraph

Posted in Software at 01:37 by graham

After a frustrating time trying to find a memory leak in my Python code with guppy / heappy hpy, I tried objgraph and, wow, it makes it so easy! Here’s what you do:

pip install objgraph

At the relevant point in your code, add an import pdb; pdb.set_trace() to drop into the debugger. Then just follow the docs on finding memory leaks with objgraph. In short you do:

>> import objgraph
>> objgraph.show_growth(limit=10)   # Start counting

Ignore that output. Call the function that leaks memory, iterate once through you loop, whatever you need to do to make your program consume more memory. Now call show_growth again:

>> my_leaky_func()
>> objgraph.show_growth(limit=10)   # Stop and show change

This time it shows the difference between now and the last time you called it. Those extra objects are the problem.

Finally you need to find where the reference to those leaky objects is being held:

>> import inspect, random
>> objgraph.show_chain(
...     objgraph.find_backref_chain(
...         random.choice(objgraph.by_type('MyBigFatObject')),
...         inspect.ismodule),
...     filename='chain.png')

That generates a lovely graph in your current directory, showing the ownership chain. Now wasn’t that easy? Thanks Marius Gedminas!

Leave a Comment

Note: Your comment will only appear on the site once I approve it manually. This can take a day or two. Thanks for taking the time to comment.