September 27, 2011

Finding memory leaks in Python with objgraph

Posted in Software at 01:37 by graham

After a frustrating time trying to find a memory leak in my Python code with guppy / heappy hpy, I tried objgraph and, wow, it makes it so easy! Here’s what you do:

pip install objgraph

At the relevant point in your code, add an import pdb; pdb.set_trace() to drop into the debugger. Then just follow the docs on finding memory leaks with objgraph. In short you do:

>> import objgraph
>> objgraph.show_growth(limit=10)   # Start counting

Ignore that output. Call the function that leaks memory, iterate once through you loop, whatever you need to do to make your program consume more memory. Now call show_growth again:

>> my_leaky_func()
>> objgraph.show_growth(limit=10)   # Stop and show change

This time it shows the difference between now and the last time you called it. Those extra objects are the problem.

Finally you need to find where the reference to those leaky objects is being held:

>> import inspect, random
>> objgraph.show_chain(
...     objgraph.find_backref_chain(
...         random.choice(objgraph.by_type('MyBigFatObject')),
...         inspect.ismodule),
...     filename='chain.png')

That generates a lovely graph in your current directory, showing the ownership chain. Now wasn’t that easy? Thanks Marius Gedminas!

September 26, 2011

DjangoCon 2011: Psychology for your webapp

Posted in Behaviour at 17:32 by graham

I got to do a 5min lightning talk at DjangoCon 2011 in Portland. The full slides are in the BarCamp post.

I’m presenting a model for applying insights from psychology to your webapp users.

Psychology for your Webapp on blip.tvscroll to 15:45 for start of my talk.

September 18, 2011

Hostage Negotiation 101

Posted in Behaviour, Society, Strategy at 21:24 by graham

I recently finished Gary Noesner’s Stalling for Time: My Life as an FBI Hostage Negotiator, by the F.B.I.‘s former head of and founder of their hostage negotiation unit. The book is a great read (and I suspect heavily ghost-written). Here’s what I learnt:

Your goal as a negotiator is to get the target(s) (the person or people you are trying to arrest) to surrender peacefully to law enforcement.

Sometimes there are hostages, and then your priority is securing their release, but usually there are not. By getting them to put down their weapons and come out you are usually saving their lives, and also protecting your colleagues.

The last resort is an armed assault by the SWAT team. Prior to negotiation being taken seriously by law enforcement, this was the only option.

Make exclusive contact

First and foremost, you need to get in contact with them. Usually they are keen to talk, and most often you can use the phone line. Sometimes you have to get the SWAT team to bring them a field telephone. Sometimes you stand outside the window or at the foot of the stairs, and shout. And occasionally, as in the Beltway sniper case you have to ask the media to say things and hope the target hears.

Read the rest of this entry »

September 16, 2011

Cleaning up old git branches

Posted in Software at 20:14 by graham

After a while working with git, you end up with lots of branches, especially if you use git-flow inspired feature branches. Here’s one way to clean them up.

For any branch, I want to know whether it has been merged, when the last commit was, and ideally if the matching ticket in our tracker has been closed.

Switch to your main branch, usually develop or master: git checkout develop

List all the branches which have been fully merged into it:

git branch -a --merged

Read the rest of this entry »

September 15, 2011

Profiling Django for CPU bound apps

Posted in Software at 20:24 by graham

For most Django apps, indeed most webapps, the bottleneck is the database. The biggest gains usually come from reducing the number of queries used, and adding database indexes. django-debug-toolbar helps a lot here. After that, caching and de-normalization also help reduce database queries.

But what if your app is CPU bound? How do you find out where it’s spending it’s time? It’s easy with the runprofileserver from django-extensions – here’s how:

Read the rest of this entry »