February 16, 2007

Jimbo – Monitor your server with Jabber

Posted in Software at 00:11 by graham

Webapps are often built and managed by single person or small teams. For example, I run 11 sites and web apps, just in my spare time, on one virtual server. I need to know if that server is up. We need a lightweight, low/zero maintenance way to monitor applications and servers. Mainly we are concerned about presence of our application. Is it there ?

Where else does presence matter ? Why, in Instant Messaging applications of course. Currently we record presence of our friends in our messenger clients. Well, servers can be our friends too !

Jabber / XMPP

XMPP is the eXtensible Messaging and Presence Protocol. Originally know as Jabber (before it was standardised), it is an open, XML-based protocol for near-real-time, extensible instant messaging and presence information. It is currently used on thousands of Jabber servers across the Internet and is used by millions of people worldwide. The Internet Engineering Task Force (IETF) formed an XMPP Working Group in 2002 to make Jabber / XMPP an official standard.

When Google built Google Talk, they chose XMPP as the core protocol.

There are free (speech & beer) servers, server software and libraries for most languages. Sounds perfect.

Brief Jabber Glossary

The Jabber HOST is the domain name of your Jabber server. For example ‘jabber.org’, ‘gmail.com’ (if using Google Talk), etc. A JID (Jabber ID) (from Wikipedia):

Every user on the network has a unique Jabber ID (usually abbreviated as JID). To avoid the need for a central server with a list of IDs, the ID is structured like an e-mail address with a username and a DNS address for the server where that user resides separated by an at sign (@). For example, if the Montague family ran a Jabber server on the domain montague.net, then the user romeo might be assigned the address romeo@montague.net.

Since a user may wish to login from multiple locations, the server allows the client to specify a further string known as a resource, which identifies which of the user’s clients it is (for example home, work and mobile). This may then be included in the JID by adding a forward slash followed by the name of the resource. For example the full JID of Romeo’s mobile would be romeo@montague.net/mobile. Messages that are simply sent to romeo@montague.net will go to all clients that Romeo has logged in at the time, but those sent to romeo@montague.net/mobile will only go to his mobile.

The RESOURCE is the last part of the JID, usually ‘Home’.

A simple monitor in Python

Python’s main XMPP library is xmppy.sourceforge.net: Send a message.

    conn = xmpp.Client(HOST)
    conn.connect()
    conn.auth(USERNAME, PASSWORD, RESOURCE)
    conn.sendInitPresence()
    conn.sendMessage( xmpp.Message(' user@jabber.org', 'Hello User') )

Replace ‘user@jabber.org’ with your jabber id, and fill in HOST, USERNAME, PASSWORD and RESOURCE. USERNAME is the JID of the jabber account you created for your machine, for example ‘my_machine_xyz@jabber.org’. RESOURCE is usually ‘Home’.

If you are using Google, replace the ‘connect’ line with:

    conn.connect( server = ('talk.google.com',5223) )

If you just want to run the monitor, JIMBo (see later) is linked at the end of this article.

The connection will automatically use SSL / TLS if it is available. The Google Talk, for example, will use it.

Next you need to allow yourself to view the machine’s presence:

    roster = conn.getRoster()
    roster.Authorize(' user@jabber.org')

Add ‘USERNAME’ as a Buddy in your Jabber IM client and you’ll know when your machine is online or offline. Do the above as a thread in a Java server, and you know when the server is running. Set an alert when your buddy ‘USERNAME’ goes offline and you have server monitoring !

You don’t even need to run your own server: There are lots of public Jabber servers (such as jabber.org), Google Talk has some, and Google Apps For Your Domain will even run one for you on your domain !

A simple bot

But wait, there’s more. Now that our machine is our friend, we probably want to chat to it:

    def onMessageReceived(conn, mess):

       sender = mess.getFrom()
       text = mess.getBody()

       print str(sender) +" send: "+ str(text)

    conn.RegisterHandler('message', onMessageReceived)

We can do anything we want on our server, remotely. We have a bot.

An aside: Jabber for you

Jabber servers have gateways to other IM’s: AIM / Yahoo / MSN / ICQ / etc. Get a Jabber id and you don’t need a multi-protocol client anymore, the gateways do it for you. With your own server, or Google Apps For Your Domain, your Jabber id can be the same as your e-mail address !

Machines Jabber the world

If we can talk to our server, so could another server. And when two machines talk to each other, enterprise types call the bit in the middle a Messaging Solution ! Consultants will want you to buy an Enterprise Service Bus, to form the basis of your new Service Oriented Architecture. Ignore all that.

Jabber has Publish / Subscribe.

Jabber / XMPP makes it easy to wire machines together. The servers and protocols have millions of users. Find an ESB that can boast that. And you can have redundancy too: ejabberd has it built in, or get your machine to add all the redundant servers into a Buddy Group (named after the service they provide for example), then cycle through all of them until it finds one that is online !

How deep is this rabbit hole ?

The more you look into Jabber, the more interesting it gest. Probably the most widely deployed Jabber server, ejabberd, the one running at jabber.org, is written in Erlang.

Erlang, developed then open-sourced in 1998 by Ericsson Computer Science Laboratory, is a soft realtime, declarative, functional language for concurrent, distributed systems. In other words, it’s not like most languages you will have used (most likely procedural and object-oriented languages), except at college if you studied Computing.

But apparently Erlang has amazing light-weight thread support.

And there’s a web server written in Erlang: YAWS (Yet Another Web Server). And when you lookup YAWS, you find this graph:

Apache and Yaws performance testing Source: http://www.sics.se/~joe/apachevsyaws.html

On the left is web server throughput in KBytes/second. Along the bottom is number of concurrent requests on a web server. In green is Apache and in red is YAWS. What this is saying is that (on the hardware of the test) Apache dies where there are over 4,000 concurrent requests. YAWS is still going at 85,000. All the detail here: Apache vs. Yaws.

So that’s pretty interesting too.

Download JIMBo

Download the Jabber Instant Monitoring BOt / JIMBo. Linux / Unix only. Installation instructions are in the zip. GPL license. v0.2.2 now includes auto-reconnect, proper daemon behaviour, and Gentoo and RedHat startup scripts

I will be presenting this at BarCamp2London this weekend

6 Comments »

  1. Micha said,

    May 11, 2010 at 08:36

    Thanks a lot for this, works great for me!

  2. Anonymus said,

    December 24, 2008 at 11:24

    Super sweet! I had to turn debugging off in xmpp/debug.py, but many thanks for these scripts!!!

  3. Jabbering from your server – O2MO 官方部落格 said,

    November 25, 2008 at 02:46

    [...] A related but more detailed code explanation (Python) is available here: Monitor your server with Jabber. [...]

  4. BarcampLondon2: Was für ein Barcamp : WebWorkBlogger said,

    July 26, 2008 at 17:51

    [...] Jimbo – Monitor your server with Jabber von Graham King hier. [...]

  5. Tim said,

    June 23, 2008 at 16:47

    For some reason my Jimbo doesn’t create a pidfile. I believe that it must be in pyDaemon that this fails to happen. Any idea how to fix it??

    Thanks

    Tim

  6. » Scaling your web application > 3 Dogs Bark said,

    February 19, 2007 at 15:10

    [...] I gave a talk at BarCampLondon2 on how to scale your web application. The talk seem to go down well and there was some good questions at the end. Overall the quality of presentations was mixed, but some particular highlights included Amazon & EC2, using Jabber to monitor your server, making web applications work offline and Metaprogramming JavaScript [...]

Leave a Comment

Note: Your comment will only appear on the site once I approve it manually. This can take a day or two. Thanks for taking the time to comment.