I believe 0MQ is great and very fast for LAN/local computations where each component runs simultaneously all the time. My system is geographically distributed where components run in arbitrary intervals. So reliable automatic reconnect and delivery was primary requirement.

Currently I’m still using Gearman on the original project I did this study for (fablistic.com), and I’m using Redis on a more recent project. I’d like to get a chance to use beanstalkd – it’s so lightweight and also a true work queue (it has ‘job completed’ semantics). I’m also loving Redis, because it does, well, everything.

(BTW, 2s2b.com is an awesome idea!)

I created a message queuing library snakeMQ http://www.snakemq.net for similar purposes. I needed a reliable communication between components in my project. My goal was to create a brokerless (no other third party components) and easy to use library with persistent queues where you don’t need to worry about connectivity. If you need pub-sub pattern then you can create a very simple broker above the library.

What solution are you using now?

David

@Abe Wow, that’s high volume, over 800 jobs / second. If that’s on a single machine, you may be hitting some OS or hardware limitation. I found an article [1] saying Digg moves 300k jobs a day through Gearman, which is nothing in comparison.

I tried emailing you, but the email you used for your comment is invalid. I’d like to know more about your setup, if you really are moving that amount of data.

[1] http://highscalability.com/blog/2009/1/13/product-gearman-open-source-message-queuing-system.html

We’ve had to death-watch queue consumers because of a particular behavior where trying to retrieve a job will block and not timeout. It seems run more reliably under quarter of the volume.

Looks pretty cool, sorry to see it didn’t make your bake-off.