On pair programming / social coding

I don’t think any developer really wanted to work alone, but they got a reputation as being loners that was unjustified because they did so much in their head.

On agile’s pace of development

I feel that the formulation of most Agile methods are a little plodding, you are coming in, you do the same amount of work every day, but you don’t have days go by where nothing gets done. Like the tortoise and the hair, and the tortoise wins the race, because the tortoise doesn’t get stuck.

On estimation

If you say what is the business need and what can we do, and if we know what we don’t know and we make a plan for that the next week, and then just work away and don’t apologise for what’s there, don’t promise things you don’t know you can do, in earnest, chip away at it.

Just know more tomorrow than you know today, and that can be about the code that you have to work with or what the real needs are. And that is the basis of anything Agile, and that can be applied anywhere.

On technical debt

If you want to go fast you take some debt. But the misunderstanding was that people thought that you never have to pay it back. If you are going to take on debt you have to pay it back or you go out of business. It’s called bankruptcy and there have been plenty of bankrupt projects.

On meta-programming

.. in something as complex and as multidimensional as product design, there is a lot of ways to approach it and sometimes if you just step back and say “This way is just a slug, I wonder if there is a shortcut going around this way”, and like I say the computer is really powerful, how can you have that computer help you in a way that most people wouldn’t think that they are allowed to program.

On the link between wiki’s and agile

if you think about [wiki] what it really does is, how can we make a database useful when the database is incomplete? Well that’s exactly the same as how can we make a computer program useful when the computer program is incomplete. Agile is all about first iteration, you make something that works, and then you add more to it, so you grow it. Well wiki is the same thing, how do I make a database of ideas in my company, that’s incomplete but still useful enough that people want to add to it?

Advice

Hang in there. I’ve devoted my life to programming, I think programming is one of the most powerful ways you can exercise your mind. I found that I can do a day job and serve whoever is paying my salary in a way that respects their goals, and still have energy left at the end of the day and do things just to please me. Or to please people who know me.

And I would say that Agile is a pretty marvellous set of values, I say open source is a different but pretty marvellous set of values and both of them are based on understanding code. So I say don’t hesitate to learn more about code, don’t hesitate to do it in an Agile way, do it test driven, maybe work with friends, but don’t hesitate to open it up.

I think that it’s easy to be overwhelmed you say “Gosh I just learnt this and now it’s that”, but I tell you, you put two days into studying something and you are already half way to expert. Stuff is coming out that fast, take the time and there is great blog posts on stuff and learn something new every week.

On CoffeeScript and the browser environment

Coffee Script and the environment will all the powerful browsers is the closest I’ve felt to the power I had twenty years ago in SmallTalk. But it’s not in one machine for one person, it’s on the whole world for the whole world. And I am excited to go down the stairs because there is a lot more to Coffee Script than I even knew and I learned it last night and I am going to go apply it and in another week I will be an expert. And you can do it too.

In summary

And we are all programmers, so let’s program.

pip install coverage
coverage run manage.py test app1 app2 app3
coverage html --include="my_project/*"
www-browser htmlcov/index.html

A disadvantage of Django’s test runner is that by default it runs tests for all the packages in INSTALLED_APPS. Usually you only want to run tests for your apps, not for third-party libraries. Hence I prefer the test runner in django-jenkins:

pip install django-jenkins

Edit settings.py to add these lines, changing “my_project” in the last line to your project’s top level package name:

INSTALLED_APPS += ('django_jenkins',)
JENKINS_TASKS = ('django_jenkins.tasks.django_tests',)
PROJECT_APPS = [appname for appname in INSTALLED_APPS if appname.startswith('my_project')]

Now run coverage with the jtest command, and it will always include all your apps, and only your apps:

coverage run manage.py jtest
coverage html --include="my_project/*"
www-browser htmlcov/index.html

As the name implies, django-jenkins is also a great choice if you’re using continuous integration server Jenkins.

The code is on github as Key-Value-Polyglot.

Each version implements enough of the get and set commands from the memcached protocol that we can test them with a memcached client.

The wonderful Internet

Thanks to a great discussion on Hacker News, in the comments here, and on reddit, I have heavily updated this post. You’re reading the new improved version.

Having the Internet review your code is humbling. It is also an amazing way to learn. The code, and my knowledge, have improved tremendously thanks to the feedback.

So far wonderful folks have contributed versions in Java, Scala, Haskell, Node.js, Ruby (event machine), Python gevent and Python diesel. If you write a version in a different language (see Spec at bottom), please send it to me, either directly or as a pull-request on github. I’ll add it to the repo.

Networking

Respond to each request with a single write. Or switch TCP_NODELAY on.

I initially thought the Go version was many times faster than the C and Python versions, but that was due to quirks in my implementation.

The C and Python versions were doing multiple small writes to respond to each “get” request. On the server side Nagle’s algorithm was buffering those small writes, waiting for an ACK from the client of it’s previous send. On the client side, Delayed ACKs meant the client wasn’t sending the needed acknowledgment until a timeout. A textbook case, assuming you know about that textbook.

The solution was to combine those multiple writes into a single one. An alternative solution would be to switch on TCP_NODELAY.

The Go version didn’t require me to know about TCP_NODELAY, which I appreciated.

Internally, Go sets the socket to be non-blocking and uses epoll and futex to wait for data. Memcached does the same thing, via libevent. I only used a single client, and the implementations are not thread-safe, so these don’t matter right now. They should matter with a large number of clients.

Writing it: Layers over Unix

Sockets: Our languages are wrapping POSIX socket system calls. C maps the calls exactly, and Python maps the C calls very closely. Go does a bit more work for us, replacing socket-bind-listen-accept with just listen-accept.

Multiple concurrent requests: Go has the magic go statement, Python has the threading library, and C has pthread_create. None of my implementations are thread safe (thanks timtadh on HN).

Read buffering: Managing the data you get from the socket was easy, once commenters taught me about C’s fdopen. Sockets are just streams of data, it’s up to the protocol to say when a message starts and ends. For the memcached protocol, we need to be able to read until \r\n, and also read a specific amount of bytes (which might include \r\n).

Go has the bufio package. C lets you treat a socket as a file, thanks to fdopen. Python does the same thing with socket.makefile, which presumably wraps fdopen.

My C is rusty, so that version took me the longest to write, and is the least legible. The biggest slowdown was because I originally implemented buffering around the socket ‘recv’ call. Commenters (thanks kbob) pointed out I didn’t need to do that, and I replaced it.

The Python version took me a while to get right, and that’s mostly my fault. I knew Python wouldn’t make me write my own buffering code, and I spent too long playing with timeout and non-blocking setups. Once I realized I could treat the socket as a file (‘makefile’) things got easier.

Python 3′s universal newline support kindly normalizes \r\n to \n, but we’re disabling it here to preserve python 2 compatibility.

The Go version was the easiest to write because I had already used the bufio package, which solves our stream buffering problem.

Read on for how to build and profile the different versions yourself.

Build

• Go version: You’ll need a recent weekly snapshot of Go (hg update weekly), or Go v1 if it is released by the time you read this. Run it with: go run memg.go, or compile it with go build memg.go and run the executable it makes.

• Python version: Uses Python 2.7 or Python 3, just run the script.

• C version: Build with gcc -std=gnu99 -g -W -Wall -pedantic -pthread -o memg memg.c. There should be no warnings.

Timing

This is explicitly not a benchmark, except if the question is “what can an average programmer do in a few hours?”.

All the versions perform about the same for me, and any differences are probably more due to my implementation than anything inherent in the languages. As antirez said in the Hacker News comments:

In order to perform such a comparison you can’t write three small throw-away programs, you need to optimize each version at your best for weeks to start to be meaningful, and you need an expert in the three systems.

To time the different versions on your machine run: time python test.py. That script requires python 2 and pylibmc (pip install pylibmc or sudo apt-get install python-pylibmc).

Try test.py against memcached too.

Profiling

To make profiling easier, each version accepts a --single command line parameter, so that it exits after a single connection.

strace was very useful for inspecting all the versions (strace -T and strace -c). Run the following for a summary of the system calls issued, and run test.py in a different window.

strace -c <the memg version you want> --single

Python

Start it in single mode, under the profiler:

python -m cProfile -o stats memg.py --single

Run test.py in a different window. View the output with pstats:

$ python
> import pstats
> p = pstats.Stats("stats")
> p.strip_dirs().sort_stats("cumulative").print_stats(20)

Go

Build an executable: go build memg.go. Run memg under prof, with --single so that it exits after the test connection closes:

go tool prof memg --single

See: prof command

You can get more detail using the runtime/pprof package and go tool pprof to examine the created file.

C

C has Valgrind, and it’s callgrind tool is very nice:

valgrind --tool=callgrind ./memg --single

Run test.py in a different window. View the output with kcachegrind:

kcachegrind callgrind.out.<num>

Spec

If you’re interested in trying it in a different language, or a different approach, here’s the short spec.

The key-value stores are an in-memory hash map. The listen on port 11211, and accept set and get commands, so that you can reproduce the following telnet interaction:

telnet 127.0.0.1 11211

set greeting 0 0 11     # User
Hello World             # User, 11 bytes
STORED                  # Server response
get greeting            # User
VALUE greeting 0 11     # Server response
Hello World             # Server response
END                     # Server response

You type the ‘set’, ‘Hello World’ and ‘get’ lines. The rest are replies. In the ‘set’ line, the ’11′ is the length of the value (‘Hello World’) in bytes. The two zero’s are ignored. All lines terminate with \r\n.

The repo has a test.py script which uses pylibmc. To get pylibmc:

• sudo pip install pylibmc
• OR sudo apt-get install python-pylibmc

If you make a version in a different language, please send it to me on github, or link it in the comments.

Go sits somewhere between C and Python. It has the static type checking and bit-twiddling powers of C (and pointers!), yet much of the speed of development and conciseness of Python (e.g. string split and join). Bruce Eckel called it a language designed to create servers, and I agree wholeheartedly.

Go is a C family language, both in the sense that it looks like C, and in the sense that you can easily call C libraries. Here is hello world:

package main;

import "fmt"

func main() {
    fmt.Println("Hello World!");
}

Interlude: Cooking up Go

This is what I think the recipe for Go looked like:

1. Go back to the 70s / 80s and find a bunch of great programmers. Say, Ken Thompson, Rob Pike, people like that. Marinate them in Bell Labs for 30 years, during which time they code in C, keep developing Unix, invent Plan 9, UTF-8, and other wonderful things.

2. Take them out, seduce them with Python, wow them with Google-scale computing. Add more amazing programmers (Brad Fitzpatrick for example), stir in Google’s near-unlimited resources.

3. Ask them how they would do C now, if they could start from scratch.

Observations

The C legacy is very strong in Go, and not just in the curly braces. For example object-orientation is done with a struct plus some methods.

Capitalization is significant. The visibility of a name outside a package is determined by whether its first character is upper case.

Multiple return values instead of exceptions. To signal an error occurred in your function you return the error, like in C, but you can return multiple values, like in Python. Error checking typically looks like this:

retval, err := myFunc(param)
if err != nil {
    // handle it
}

The style guide is whatever program gofmt says it is. It will often just fix it for you. Use whatever style you want, then run it through gofmt before checking-in.

Interfaces, with no explicit implementations. If you have the methods of that interface, you count. That gives statically checked duck typing. Lovely.

Go has arrays, but you usually use slices instead. A slice works like a Python slice, but internally it’s a smart pointer to an array, and has a different type (which matters because we’re doing static typing remember). The array is it’s backing storage. I found this tricky to understand at first.

Everything is passed by value: Objects (structs) and Arrays are copied before being passed. For arrays it doesn’t matter because you’re usually passing slices (which wrap pointers – as do maps). When defining your own types, you have to remember to always pass a pointer.

The compiler is strict: Unused variables are a compiler error.

Negatives

The one big potential negative is that it’s compiled. In Python I often find myself stepping into third party libraries. Always having the source code is fantastic. One print statement deep in a someone else’s code can really help comprehension. No being able to do this in Java was a problem. I don’t have enough experience with Go to say if this will be a problem during debugging or not.

A read-eval-print loop would have made learning Go much easier. I can’t overstate how often I try out simple commands, like date formatting, in ipython. In Java I used beanshell.

You can’t daemonize, because of the way go-routines work. This could be fixed soon, see Russ Cox’s message here. In practice I usually use Upstart, so it doesn’t cause a problem.

Go has an un-googlable name. The convention is to use golang as a search term.

The language is still very young and in flux. See for example the GitHub network for gocurse. v1 of Go is due in the first half of this year (2012), so hopefully things will settle down.

The extremely useful string manipulation methods are all in the strings package, instead of being methods on the string object. There are objects in many packages, so there is no doubt a very good reason for string not having these methods. That’s the thing about Go, there’s usually a very good reason for everything it does. Learning Go is an education. Oh, that’s a positive.

Positives

Old programs read like quiet conversations between a well-spoken research worker and a well-studied mechanical colleague, not as a debate with a compiler – Dick Gabriel.

Russ Cox uses that Dick Gabriel quote to illustrate what they are aiming for with Go. In some hard-to-define way, Go feels serious and grown up. It has high expectations of you.

A familiar language. As a Python programmer with a Java and C background, I was productive very fast.

A modern language. Go has everything you’d expect a modern language to have: Unicode, garbage collection, multi-core (GOMAXPROCS), built-in map, string and array types, closures, unit-testing , an official style guide (‘gofmt’), reflection, etc

The documentation is excellent, and usually available from the command line: godoc <package>.

The compiler is very fast, fast enough that you can use Go as a scripting language. Use gorun as your hash-bang line.

Static typing makes programming easier. That is a debatable point, but it does for me. Static typing makes code easier to read, and make me feel a little more comfortable that things won’t go boom in the middle of the night.

Concurrency

The one big NEW (to me at least) thing Go does very well is concurrency. The keyword go starts a new go-routine. That’s what you use instead of sub-processes, threads, or co-routines. The go-routine might run on the same core / processor as your current one, or it might not. You don’t worry about that.

Go-routine’s communicate on channels. You epoll those channels using the select keyword.

Here is the heart of hatcog, my IRC client. A go-routine listens on the port connected to the server and puts incoming data on channel fromServer. Another go-routine listens for keyboard input, and puts that on channel fromUser. The middle bit watches both channels:

for self.isRunning {

    select {
    case serverData, ok = <-fromServer:
        if ok {
            self.onServer(serverData)
        } else {
            self.isRunning = false
        }

    case userInput, ok = <-fromUser:
        if ok {
            self.onUser(userInput)
        } else {
            self.isRunning = false
        }
    }

}

More details on Go’s concurrency model

Do you see how that makes servers so much easier? You could write a webapp that’s it’s own server. The “C10K problem” might not be a problem at all.

Conclusion

Would I be happy working with Go as my main language? Yes, I would. It’s a joy to work with, and I got productive very fast.

Am I using it instead of Python for all my new projects? No, I’m not. There are two reasons for that. Firstly, it’s a very young language, so library availability is limited (for example, I need curses). Second, well, Python is just so amazing.

If I was still doing Java, or (heaven forbid) C++, I would invest heavily in Go. It was designed to replace them, and it does that well.

As a Python guy, the summary is more nuanced. I see the benefit of static typing. Other claimed benefits of Go over Python are that it’s faster, and that it’s “better at scale”.

For some things I’ve done Go has been faster, for other things Python. The biggest difference is probably in how well I write the code. Although I love speed in principle, nothing I do is CPU bound.

The “better at scale” argument doesn’t really apply to what I do for a living, which is building webapps with Django. We scale performance-wise by adding servers, and code-wise by adding small self-contained ‘apps’ (in the Django sense).

Go’s sweet spot is building servers. It makes concurrency safer and easier than the current options. I’m going to keep using it for that. And because it’s fun.

Next you might want to page through this Go presentation, or dive in to the (rather academic) tutorial.

(Did I make a mistake? Please correct me in the comments. Thanks!)

A year here and he still dreamed of cyberspace, hope fading nightly. All the speed he took, all the turns he’d taken and the corners he’d cut in Night City, and still he’d see the matrix in his sleep, bright lattices of logic unfolding across that colorless void… The Sprawl was a long strange way home over the Pacific now, and he was no console man, no cyberspace cowboy. Just another hustler, trying to make it through.

But the dreams came on in the Japanese night like livewire voodoo, and he’d cry for it, cry in his sleep, and wake alone in the dark, curled in his capsule in some coffin hotel, his hands clawed into the bedslab, temperfoam bunched between his fingers, trying to reach the console that wasn’t there.

From Neuromancer, p4-5

1. Establish what “normal” looks like in your world. If your story is set in present-day New York, you can do that quickly, during the opening credits. If your story is set in Middle Earth, it takes a lot longer.
2. Break the routine. Frodo has to leave the shire. This is when the story really starts, and why you’re watching it. Today is different.

The remake of the classic 80s series, Miami Vice (2006), fails at number 2. For the whole film, two undercover vice squad detectives go undercover to bust a vice gang. Sure there’s fast cars, guns, all that, but it could of been so much better.

About fifteen minutes into the movie, Sonny and Ricardo (the protagonists) are talking to an informer by the side of the motorway. The informer walks in front of a truck and dies right in front of the detectives. In any real police force they would have immediately been put on sick leave, with counselling. That’s a break in the routine, and that’s the movie I want to see.

You could make it very dark. One of them goes back to work in a few days, leaves the story. The other, the focus, stays on sick leave. He realises he’s spent his whole live at work, often pretending to be someone else. No family, no friends, no community, no interests, and a very uncertain sense of identity. He mows his lawn. He slowly falls apart.

But it needn’t be dark. Our hero mows the lawn, and gets to know the lives of people on his street. He gets caught up in an intrigue that uses his elite vice-squadding skills. Something much below his usual level of investigation, but something that matters to this community. Maybe the old ladies’ house is going to be foreclosed by the bank. Whatever the plot device is, it allows our protagonist to change, to gain a new perspective on life.

A gardener going undercover to bust a crime ring is an interesting story. An undercover cop going undercover, isn’t. Stories are interesting when they change the people in them.

// Is given File a terminal?
func isatty(file *os.File) bool {
    stat, _ := file.Stat()
    return !stat.IsFifo()
}

if ! isatty(os.Stdin) {
  // read what was piped in
}

For example:

$ ./myprog                # isatty = yes
$ echo "boo" | ./myprog   # isatty = no

Mirrors the behaviour of Python’s isatty.

Disclaimer: I got this working last night, so no promises. You’ll certainly want to tweak haproxy’s config for performance. I also only tested it with socket.io’s web socket transport.

Overview

• stunnel decrypts the ssl, so everything after that doesn’t know about it. It decrypts both web traffic (HTTPS to HTTP), and web socket traffic (WSS to WS).
• haproxy sends web socket traffic to node and web traffic to nginx.
• node runs socket.io, handling the web socket traffic.
• nginx serves static content.
• gunicorn runs python / django, and there’s a database out back somewhere, but that’s not relevant here.

Currently nginx doesn’t support HTTP/1.1 for it’s backends, so it can’t proxy web socket traffic. That’s why we have haproxy.
But haproxy doesn’t do SSL, that’s why we have stunnel.
And haproxy isn’t a web server, so we still need nginx.

Generate a self-signed cert

To test this you’ll need an SSL certificate. Here’s how (thanks Victor Farazdagi):

openssl genrsa -out mysite.key 1024
openssl req -new -key mysite.key -out mysite.csr  # common name == your domain
openssl x509 -req -days 365 -in mysite.csr -signkey mysite.key -out mysite.crt

stunnel

Install: sudo apt-get install stunnel4.
Enable it by editing /etc/default/stunnel and settings ENABLED=1.

Config: /etc/stunnel/stunnel.conf

cert = /etc/stunnel/localhost.crt
key = /etc/stunnel/localhost.key

debug = 5
output = /var/log/stunnel4/stunnel.log

[https]
accept = 443
connect = 81
TIMEOUTclose = 0

haproxy

Install: sudo apt-get install haproxy

Config: /etc/haproxy/haproxy.cfg

global
    maxconn 4096
    daemon

defaults
    mode http
    log 127.0.0.1 local1 debug
    option httplog

frontend all 0.0.0.0:81
    timeout client 86400000
    default_backend www_backend
    acl is_websocket hdr(Upgrade) -i WebSocket
    acl is_websocket path_beg /socket.io/

    use_backend socket_backend if is_websocket

backend www_backend
    balance roundrobin
    option forwardfor # This sets X-Forwarded-For
    option httpclose
    timeout server 30000
    timeout connect 4000
    server server1 localhost:82 weight 1 maxconn 1024 check

backend socket_backend
    balance roundrobin
    option forwardfor # This sets X-Forwarded-For
    option httpclose
    timeout queue 5000
    timeout server 86400000
    timeout connect 86400000
    server server1 localhost:9000 weight 1 maxconn 1024 check

haproxy logs to syslog, and expects it to be in server mode, so you need to set that up too (thanks Kevin van Zonneveld):

rsyslog config: /etc/rsyslog.d/haproxy.conf

$ModLoad imudp
$UDPServerRun 514
$UDPServerAddress 127.0.0.1

local1.* -/var/log/haproxy_1.log
& ~ 

Then bounce rsyslog: sudo restart rsyslog

nginx

First bounce http traffic to https: /etc/nginx/sites-enabled/default

server {
  listen 80;
  server_name _; # Catch requests that don't match any other server name
  rewrite ^ https://myapp.example.com$request_uri? permanent;
}

Next setup nginx on port 82, and make sure to rewrite Location responses (see THIS ONE below):

server {
  listen 82;
  server_name myapp.example.com;

  location /static {
    root /var/www/myapp.example.com;
  }

  location / {
    proxy_pass http://unix:/tmp/ginger-gunicorn.sock;
    ## THIS ONE ##
    proxy_redirect http://myapp.example.com https://myapp.example.com;
    ## END THIS ONE ##
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  }
}

Node

Put node on port 9000, with a standard config. Make sure to ask the client library to connect securely, so that it stays on port 443 (https then wss):

var socket = io.connect('https://myapp.example.com', {secure: true})

Good luck

This setup is working for me, so far. There’s quite a few moving parts. HTTP/1.1 is coming to nginx (it’s in the dev version already), so hopefully we’ll be able to use that instead of haproxy and stunnel soon.

The examples are in Python, with a summary example in Go (golang) at the end.

Single line with \r (carriage return)

Instead of printing a \n (which most ‘print’ methods do by default), print a \r. That sends the cursor back to the beginning of the current line (carriage return), without dropping down to a new line (line feed).

import time, sys
total = 10
for i in range(total):
    sys.stdout.write('%d / %d\r' % (i, total))
    sys.stdout.flush()
    time.sleep(0.5)
print('Done     ')

We do sys.stdout.flush() to flush the output, otherwise the operating system buffers it and you don’t see anything until the end. Note also the extra spaces after ‘Done’, to clear previous output. Remove them and you’ll see why they are needed.

ANSI escape sequences

ANSI escape sequences are special non-printable characters you send to the terminal to control it. When you do ls --color, ls is using ANSI escape sequences to pretty things up for you.

When the terminal receives an ESC character (octal 33 – see man ascii) it expects control information to follow, instead of printable data. Most escape sequences start with ESC and [.

Bold

To display something in bold, switch on the bright attribute (1), then reset it afterwards (0).

def bold(msg):
    return u'\033[1m%s\033[0m' % msg

print('This is %s yes it is' % bold('bold'))

Color

There are 8 colors, ANSI codes 30 to 37, which can have the bold modifier, making 16 colors. Colors are surprisingly consistent across terminals, but can usually be changed by the user.

def color(this_color, string):
    return "\033[" + this_color + "m" + string + "\033[0m"

for i in range(30, 38):
    c = str(i)
    print('This is %s' % color(c, 'color ' + c))

    c = '1;' + str(i)
    print('This is %s' % color(c, 'color ' + c))

If that's not enough for you, you can have 256 colors (thanks lucentbeing), but only if your terminal is set to 256 color mode. You usually get 256 colors by putting one of these lines in your .bashrc:

export TERM='screen-256color'   # Use this is you use tmux or screen. Top choice!
export TERM='xterm-256color'    # Use this otherwise

Here's lots of colors (copy color function from above):

for i in range(256):
    c = '38;05;%d' % i
    print( color(c, 'color ' + c) )

Clear the screen

A pair of ANSI escape sequences:

import sys

def clear():
    """Clear screen, return cursor to top left"""
    sys.stdout.write('\033[2J')
    sys.stdout.write('\033[H')
    sys.stdout.flush()

clear()

More ANSI power

You can do lots more with ANSI escape sequences, such as change the background color, move the cursor around, and even make it blink. Use your power wisely.

Measure the screen

You can find out how many characters wide or lines high the current terminal is. From the command line, try tput lines or tput cols. To get the data direct from Unix:

import sys
import fcntl
import termios
import struct

lines, cols = struct.unpack('hh',  fcntl.ioctl(sys.stdout, termios.TIOCGWINSZ, '1234'))
print('Terminal is %d lines high by %d chars wide' % (lines, cols))

This is making a system call, hence it's a bit inscrutable. It's requesting TIOCGWINSZ information, for file sys.stdout (the terminal), and then interprets it as two packed short integers (hh). The '1234' is just placeholder, which must be four chars long.

A pretty progress bar

Putting some of the above together, here's a progress bar.

import sys
import fcntl
import termios
import struct
import time

COLS = struct.unpack('hh',  fcntl.ioctl(sys.stdout, termios.TIOCGWINSZ, '1234'))[1]

def bold(msg):
    return u'\033[1m%s\033[0m' % msg

def progress(current, total):
    prefix = '%d / %d' % (current, total)
    bar_start = ' ['
    bar_end = '] '

    bar_size = COLS - len(prefix + bar_start + bar_end)
    amount = int(current / (total / float(bar_size)))
    remain = bar_size - amount

    bar = 'X' * amount + ' ' * remain
    return bold(prefix) + bar_start + bar + bar_end

NUM = 100
for i in range(NUM + 1):
    sys.stdout.write(progress(i, NUM) + '\r')
    sys.stdout.flush()
    time.sleep(0.05)
print('\n')

That's it! Happy console scripts.

And now the progress bar in Go golang, for fun

package main

import (
    "time"
    "os"
    "strconv"
    "strings"
    "syscall"
    "unsafe"
)

const (
    ONE_MSEC    = 1000 * 1000
    _TIOCGWINSZ = 0x5413    // On OSX use 1074295912. Thanks zeebo
    NUM         = 50
)

func main() {

    var bar string

    cols := TerminalWidth()

    for i := 1; i <= NUM; i++ {
        bar = progress(i, NUM, cols)
        os.Stdout.Write([]byte(bar + "\r"))
        os.Stdout.Sync()
        time.Sleep(ONE_MSEC * 50)
    }
    os.Stdout.Write([]byte("\n"))
}

func Bold(str string) string {
    return "\033[1m" + str + "\033[0m"
}

func TerminalWidth() int {
    sizeobj, _ := GetWinsize()
    return int(sizeobj.Col)
}

func progress(current, total, cols int) string {
    prefix := strconv.Itoa(current) + " / " + strconv.Itoa(total)
    bar_start := " ["
    bar_end := "] "

    bar_size := cols - len(prefix + bar_start + bar_end)
    amount := int(float32(current) / (float32(total) / float32(bar_size)))
    remain := bar_size - amount

    bar := strings.Repeat("X", amount) + strings.Repeat(" ", remain)
    return Bold(prefix) + bar_start + bar + bar_end
}

type winsize struct {
    Row    uint16
    Col    uint16
    Xpixel uint16
    Ypixel uint16
}

func GetWinsize() (*winsize, os.Error) {
    ws := new(winsize)

    r1, _, errno := syscall.Syscall(syscall.SYS_IOCTL,
        uintptr(syscall.Stdin),
        uintptr(_TIOCGWINSZ),
        uintptr(unsafe.Pointer(ws)),
    )

    if int(r1) == -1 {
        return nil, os.NewSyscallError("GetWinsize", int(errno))
    }
    return ws, nil
}

Dhanji R. Prasanna has an excellent retrospective on his time on the Google Wave team. He sums up the problem with big teams very well:

And this is the essential broader point–as a programmer you must have a series of wins, every single day. It is the Deus Ex Machina of hacker success. It is what makes you eager for the next feature, and the next after that. And a large team is poison to small wins. The nature of large teams is such that even when you do have wins, they come after long, tiresome and disproportionately many hurdles. And this takes all the wind out of them.

For me, that’s really the crux of it. As a programmer, it kills you to not get stuff done. Large teams necessarily involve more communication, more complexity, and less getting stuff done. Large teams are a programmers equivalent of retirement.