// Is given File a terminal?
func isatty(file *os.File) bool {
    stat, _ := file.Stat()
    return !stat.IsFifo()
}

if ! isatty(os.Stdin) {
  // read what was piped in
}

For example:

$ ./myprog                # isatty = yes
$ echo "boo" | ./myprog   # isatty = no

Mirrors the behaviour of Python’s isatty.

Disclaimer: I got this working last night, so no promises. You’ll certainly want to tweak haproxy’s config for performance. I also only tested it with socket.io’s web socket transport.

Overview

• stunnel decrypts the ssl, so everything after that doesn’t know about it. It decrypts both web traffic (HTTPS to HTTP), and web socket traffic (WSS to WS).
• haproxy sends web socket traffic to node and web traffic to nginx.
• node runs socket.io, handling the web socket traffic.
• nginx serves static content.
• gunicorn runs python / django, and there’s a database out back somewhere, but that’s not relevant here.

Currently nginx doesn’t support HTTP/1.1 for it’s backends, so it can’t proxy web socket traffic. That’s why we have haproxy.
But haproxy doesn’t do SSL, that’s why we have stunnel.
And haproxy isn’t a web server, so we still need nginx.

Generate a self-signed cert

To test this you’ll need an SSL certificate. Here’s how (thanks Victor Farazdagi):

openssl genrsa -out mysite.key 1024
openssl req -new -key mysite.key -out mysite.csr  # common name == your domain
openssl x509 -req -days 365 -in mysite.csr -signkey mysite.key -out mysite.crt

stunnel

Install: sudo apt-get install stunnel4.
Enable it by editing /etc/default/stunnel and settings ENABLED=1.

Config: /etc/stunnel/stunnel.conf

cert = /etc/stunnel/localhost.crt
key = /etc/stunnel/localhost.key

debug = 5
output = /var/log/stunnel4/stunnel.log

[https]
accept = 443
connect = 81
TIMEOUTclose = 0

haproxy

Install: sudo apt-get install haproxy

Config: /etc/haproxy/haproxy.cfg

global
    maxconn 4096
    daemon

defaults
    mode http
    log 127.0.0.1 local1 debug
    option httplog

frontend all 0.0.0.0:81
    timeout client 86400000
    default_backend www_backend
    acl is_websocket hdr(Upgrade) -i WebSocket
    acl is_websocket path_beg /socket.io/

    use_backend socket_backend if is_websocket

backend www_backend
    balance roundrobin
    option forwardfor # This sets X-Forwarded-For
    option httpclose
    timeout server 30000
    timeout connect 4000
    server server1 localhost:82 weight 1 maxconn 1024 check

backend socket_backend
    balance roundrobin
    option forwardfor # This sets X-Forwarded-For
    option httpclose
    timeout queue 5000
    timeout server 86400000
    timeout connect 86400000
    server server1 localhost:9000 weight 1 maxconn 1024 check

haproxy logs to syslog, and expects it to be in server mode, so you need to set that up too (thanks Kevin van Zonneveld):

rsyslog config: /etc/rsyslog.d/haproxy.conf

$ModLoad imudp
$UDPServerRun 514
$UDPServerAddress 127.0.0.1

local1.* -/var/log/haproxy_1.log
& ~ 

Then bounce rsyslog: sudo restart rsyslog

nginx

First bounce http traffic to https: /etc/nginx/sites-enabled/default

server {
  listen 80;
  server_name _; # Catch requests that don't match any other server name
  rewrite ^ https://myapp.example.com$request_uri? permanent;
}

Next setup nginx on port 82, and make sure to rewrite Location responses (see THIS ONE below):

server {
  listen 82;
  server_name myapp.example.com;

  location /static {
    root /var/www/myapp.example.com;
  }

  location / {
    proxy_pass http://unix:/tmp/ginger-gunicorn.sock;
    ## THIS ONE ##
    proxy_redirect http://myapp.example.com https://myapp.example.com;
    ## END THIS ONE ##
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  }
}

Node

Put node on port 9000, with a standard config. Make sure to ask the client library to connect securely, so that it stays on port 443 (https then wss):

var socket = io.connect('https://myapp.example.com', {secure: true})

Good luck

This setup is working for me, so far. There’s quite a few moving parts. HTTP/1.1 is coming to nginx (it’s in the dev version already), so hopefully we’ll be able to use that instead of haproxy and stunnel soon.

The examples are in Python, with a summary example in Go (golang) at the end.

Single line with \r (carriage return)

Instead of printing a \n (which most ‘print’ methods do by default), print a \r. That sends the cursor back to the beginning of the current line (carriage return), without dropping down to a new line (line feed).

import time, sys
total = 10
for i in range(total):
    sys.stdout.write('%d / %d\r' % (i, total))
    sys.stdout.flush()
    time.sleep(0.5)
print('Done     ')

We do sys.stdout.flush() to flush the output, otherwise the operating system buffers it and you don’t see anything until the end. Note also the extra spaces after ‘Done’, to clear previous output. Remove them and you’ll see why they are needed.

ANSI escape sequences

ANSI escape sequences are special non-printable characters you send to the terminal to control it. When you do ls --color, ls is using ANSI escape sequences to pretty things up for you.

When the terminal receives an ESC character (octal 33 – see man ascii) it expects control information to follow, instead of printable data. Most escape sequences start with ESC and [.

Bold

To display something in bold, switch on the bright attribute (1), then reset it afterwards (0).

def bold(msg):
    return u'\033[1m%s\033[0m' % msg

print('This is %s yes it is' % bold('bold'))

Color

There are 8 colors, ANSI codes 30 to 37, which can have the bold modifier, making 16 colors. Colors are surprisingly consistent across terminals, but can usually be changed by the user.

def color(this_color, string):
    return "\033[" + this_color + "m" + string + "\033[0m"

for i in range(30, 38):
    c = str(i)
    print('This is %s' % color(c, 'color ' + c))

    c = '1;' + str(i)
    print('This is %s' % color(c, 'color ' + c))

If that's not enough for you, you can have 256 colors (thanks lucentbeing), but only if your terminal is set to 256 color mode. You usually get 256 colors by putting one of these lines in your .bashrc:

export TERM='screen-256color'   # Use this is you use tmux or screen. Top choice!
export TERM='xterm-256color'    # Use this otherwise

Here's lots of colors (copy color function from above):

for i in range(256):
    c = '38;05;%d' % i
    print( color(c, 'color ' + c) )

Clear the screen

A pair of ANSI escape sequences:

import sys

def clear():
    """Clear screen, return cursor to top left"""
    sys.stdout.write('\033[2J')
    sys.stdout.write('\033[H')
    sys.stdout.flush()

clear()

More ANSI power

You can do lots more with ANSI escape sequences, such as change the background color, move the cursor around, and even make it blink. Use your power wisely.

Measure the screen

You can find out how many characters wide or lines high the current terminal is. From the command line, try tput lines or tput cols. To get the data direct from Unix:

import sys
import fcntl
import termios
import struct

lines, cols = struct.unpack('hh',  fcntl.ioctl(sys.stdout, termios.TIOCGWINSZ, '1234'))
print('Terminal is %d lines high by %d chars wide' % (lines, cols))

This is making a system call, hence it's a bit inscrutable. It's requesting TIOCGWINSZ information, for file sys.stdout (the terminal), and then interprets it as two packed short integers (hh). The '1234' is just placeholder, which must be four chars long.

A pretty progress bar

Putting some of the above together, here's a progress bar.

import sys
import fcntl
import termios
import struct
import time

COLS = struct.unpack('hh',  fcntl.ioctl(sys.stdout, termios.TIOCGWINSZ, '1234'))[1]

def bold(msg):
    return u'\033[1m%s\033[0m' % msg

def progress(current, total):
    prefix = '%d / %d' % (current, total)
    bar_start = ' ['
    bar_end = '] '

    bar_size = COLS - len(prefix + bar_start + bar_end)
    amount = int(current / (total / float(bar_size)))
    remain = bar_size - amount

    bar = 'X' * amount + ' ' * remain
    return bold(prefix) + bar_start + bar + bar_end

NUM = 100
for i in range(NUM + 1):
    sys.stdout.write(progress(i, NUM) + '\r')
    sys.stdout.flush()
    time.sleep(0.05)
print('\n')

That's it! Happy console scripts.

And now the progress bar in Go golang, for fun

package main

import (
    "time"
    "os"
    "strconv"
    "strings"
    "syscall"
    "unsafe"
)

const (
    ONE_MSEC    = 1000 * 1000
    _TIOCGWINSZ = 0x5413    // On OSX use 1074295912. Thanks zeebo
    NUM         = 50
)

func main() {

    var bar string

    cols := TerminalWidth()

    for i := 1; i <= NUM; i++ {
        bar = progress(i, NUM, cols)
        os.Stdout.Write([]byte(bar + "\r"))
        os.Stdout.Sync()
        time.Sleep(ONE_MSEC * 50)
    }
    os.Stdout.Write([]byte("\n"))
}

func Bold(str string) string {
    return "\033[1m" + str + "\033[0m"
}

func TerminalWidth() int {
    sizeobj, _ := GetWinsize()
    return int(sizeobj.Col)
}

func progress(current, total, cols int) string {
    prefix := strconv.Itoa(current) + " / " + strconv.Itoa(total)
    bar_start := " ["
    bar_end := "] "

    bar_size := cols - len(prefix + bar_start + bar_end)
    amount := int(float32(current) / (float32(total) / float32(bar_size)))
    remain := bar_size - amount

    bar := strings.Repeat("X", amount) + strings.Repeat(" ", remain)
    return Bold(prefix) + bar_start + bar + bar_end
}

type winsize struct {
    Row    uint16
    Col    uint16
    Xpixel uint16
    Ypixel uint16
}

func GetWinsize() (*winsize, os.Error) {
    ws := new(winsize)

    r1, _, errno := syscall.Syscall(syscall.SYS_IOCTL,
        uintptr(syscall.Stdin),
        uintptr(_TIOCGWINSZ),
        uintptr(unsafe.Pointer(ws)),
    )

    if int(r1) == -1 {
        return nil, os.NewSyscallError("GetWinsize", int(errno))
    }
    return ws, nil
}

Dhanji R. Prasanna has an excellent retrospective on his time on the Google Wave team. He sums up the problem with big teams very well:

And this is the essential broader point–as a programmer you must have a series of wins, every single day. It is the Deus Ex Machina of hacker success. It is what makes you eager for the next feature, and the next after that. And a large team is poison to small wins. The nature of large teams is such that even when you do have wins, they come after long, tiresome and disproportionately many hurdles. And this takes all the wind out of them.

For me, that’s really the crux of it. As a programmer, it kills you to not get stuff done. Large teams necessarily involve more communication, more complexity, and less getting stuff done. Large teams are a programmers equivalent of retirement.

An article on monitoring iPhone traffic by Craig Dunn got me wondering what the iPad is sending over the wire. That led me to blocking many of the adverts apps show. Here’s how.

1. Setup a proxy (squid on ubuntu)

First you need to setup a proxy, and send all your iPad’s network traffic through that. On Ubuntu squid is easy to setup: sudo apt-get install squid

Edit the config file in /etc/squid/squid.conf and comment in or out the following lines:

# Comment in - change if your local IP's don't start with 192.168
acl localnet src 192.168.0.0/16

# Comment out
#http_access deny CONNECT !SSL_ports

# Comment in
http_access allow localnet

# Make sure you have this line
access_log /var/log/squid/access.log squid

# Add this line in the log_fqdn section
log_fqdn on

# Set to off
strip_query_terms off

# We'll play with this later
hosts_file /etc/hosts

Restart squid: sudo restart squid. Tail the log file: sudo tail -f /var/log/squid/access.log

2. Set your iPad to use the proxy

On your iPad go into Settings then Wi-Fi, and tap the white / blue arrow by your network. In the HTTP Proxy section switch to Manual and type in the local IP address (192.168…) of your machine running squid, and port 3128 (squid’s default).

Open an app. You should see some traffic come through in your tail of the log. Not all apps use the network, but most do.

3. View the POST requests (tcpdump)

Your squid log will show the GET requests, but not the data of the POST request. You’re probably seeing lots of this: POST http://data.flurry.com/aap.do. Flurry seems to be the DoubleClick of iOS.

To view the POST contents, you need tcpdump: sudo apt-get install tcpdump. Then monitor TCP packets on port 80:

sudo tcpdump -i eth0 -A 'tcp port 80 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'

(I copied that line from tcpdump’s man page)

Examples

Here’s what some of the apps on my iPad are sending

Angry Birds

POST http://data.flurry.com/aap.do - DIRECT/216.74.41.14 application/octet-stream
GET http://svrsecure-g2-aia.verisign.com/SVRSecureG2.cer - NONE/- text/plain
GET http://ax.init.itunes.apple.com/bag.xml? - DIRECT/63.80.4.50 text/xml
CONNECT p17-buy.itunes.apple.com:443 - DIRECT/17.154.66.17 -

Talking Tom

POST http://data.flurry.com/aap.do - DIRECT/216.74.41.14 application/octet-stream
GET http://outfit7-affirmations.appspot.com/rest/v1/news/ipad/talking/en/? - DIRECT/74.125.127.141 application/json
POST http://outfit7-affirmations.appspot.com/rest/talkingFriends/v2/iPad/? - DIRECT/74.125.127.141 application/json
GET http://apps.outfit7.com/static/icon/news/IconNewsFreeA72.png - DIRECT/74.125.127.121 image/png
GET http://apps.outfit7.com/static/buttons/rect/news/RectWhite120.png? - DIRECT/74.125.127.121 image/png
GET http://apps.outfit7.com/rest/talkingFriends/v1/push-notification/put/iPad/? - DIRECT/74.125.127.121 application/json
GET http://apps.outfit7.com/static/icon/news/IconNewsFreeA72.png - DIRECT/74.125.127.121 image/png
GET http://apps.outfit7.com/static/buttons/rect/news/RectWhite120.png? - DIRECT/74.125.127.121 image/png
GET http://apps.outfit7.com/rest/talkingFriends/v1/push-notification/put/iPad/? - DIRECT/74.125.127.121 application/json
GET http://apps.outfit7.com/ad/ad.jsp? - DIRECT/74.125.127.121 application/json
POST http://i.w.inmobi.com/showad.asm? - DIRECT/174.140.140.33 text/html
GET http://www.gstatic.com/afma/sdk-core-v40.js - DIRECT/74.125.127.120 text/javascript
GET http://googleads.g.doubleclick.net/mads/gma? - DIRECT/74.125.127.154 text/html
GET http://media.admob.com/gmsg.js - DIRECT/74.125.127.120 text/javascript
GET http://mmv.admob.com/p/i/4d/59/4d59f1cc8bcdb1e30dfa346b08195126-728x90.png - DIRECT/63.80.242.35 image/png

Here’s Talking Tom’s POST request to flurry.

For some reason data.flurry.com’s IP reverses to perureisen.com, a seemingly unrelated travel site. Maybe they share some virtual hosting.

turok is the name of my machine running the proxy. Notice the string IPHONEf0842dbXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXbce (I obscured part of it). That seems to be a unique id for my iPad, which all post’s to flurry share, and is presumably used to track me across apps.

Here’s the POST:

turok.local.39475 > perureisen.com.www
POST /aap.do HTTP/1.0
Host: data.flurry.com
User-Agent: TalkingCatIpad/1.7 CFNetwork/485.13.9 Darwin/11.0.0
Content-Type: application/octet-stream
Accept: */*
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Pragma: no-cache
Content-Length: 501
Via: 1.1 turok:3128 (squid/2.7.STABLE9)
X-Forwarded-For: 192.168.1.79
Cache-Control: max-age=259200
Connection: keep-alive

turok.local.39475 > perureisen.com.www
E...h.@.@......@.J)..3.P....j.G7P.9.W....................0...3......R8THSKSDBS4YZQTXWLDP..1.7....IPHONEf0842dbXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXbce...1..z....
3........device.model.1..iPad1,1......1.7...3.._.........GridPageShown.....mericaUITouches......AppLaunched........AppLaunched....launchedVia..
GridPageShown....group..TalkingFriends.................GridClose.........f......... UITouches....touches..head.......4.................

Some of these POST’s to data.flurry.com include my location. The response is just an empty 200.

Most of the apps’s network traffic looks like those two, although sometimes they do weird things. An app called My Horse does this:

My Horse

POST http://data.flurry.com/aap.do - NONE/- text/html
GET http://ax.init.itunes.apple.com/bag.xml?ix=2 - DIRECT/63.80.4.24 text/xml
GET http://itunes.apple.com/WebObjects/MZStore.woa/wa/fetchSoftwareAddOns?appAdamId=421167112&bvrs=1.1.2&sfId=143441-1,9&offerNames=com.naturalmotion.h0rs3.110Gems,com.naturalmotion.h0rs3.1200Gems,com.naturalmotion.h0rs3.20Gems,com.naturalmotion.h0rs3.224Gems,com.naturalmotion.h0rs3.2500Gems,com.naturalmotion.h0rs3.42Gems,com.naturalmotion.h0rs3.450Gems&bid=com.naturalmotion.h0rs3&icuLocale=en_US&appExtVrsId=4160270 - DIRECT/63.80.4.65 text/xml
GET http://svrsecure-g2-aia.verisign.com/SVRSecureG2.cer - NONE/- text/plain
CONNECT p17-buy.itunes.apple.com:443 - DIRECT/17.154.66.17 -
PUT http://mf-horse-upload-rel.s3.amazonaws.com/audit/uid=1345522/directoryListing.txt - DIRECT/207.171.189.81 -
PUT http://mf-horse-upload-rel.s3.amazonaws.com/logFiles/uid=1345522/Fri_Sep_23_12:58:26_2011_144189.txt - DIRECT/207.171.189.81 -
PUT http://mf-horse-upload-rel.s3.amazonaws.com/logFiles/uid=1345522/Fri_Sep_23_16:30:00_2011_375253.txt - DIRECT/207.171.189.81 -
PUT http://mf-horse-upload-rel.s3.amazonaws.com/logFiles/uid=1345522/Sat_Oct__1_17:26:19_2011_317371.txt - DIRECT/207.171.189.81 -
[snip - lots and lots more logFiles PUTing]

Pretty weird huh? I suspect someone forgot to remove their debugging code, or else they are being exceptionally nosy.

Ad Blocking, privacy enhancement

You’ll notice a few apps hitting ad servers, then downloading advert images. You probably also don’t want all that monitoring nonsense. The best way to block those requests would be to keep running your iPad through a proxy, and use /etc/hosts to set relevant sites to 127.0.0.1. That requires a local PC running all the time, so instead I used a setting on my router to block certain sites.

Here’s a list of sites I’m blocking:

• data.flurry.com
• mob.adwhirl.com
• media.admob.com
• mmv.admob.com
• a.admob.com
• i.w.inmobi.com
• iphone.playhaven.com
• ingameads.gameloft.com
• ads.mobclix.com
• s.mobclix.com
• thisadworks.com
• c33.smaato.net
• c26.smaato.net
• i.xx.openx.com
• ads.mp.mydas.mobi

All apps still work fine. On one I get the words ‘Request failed’ where the advert usually appears.

Some of the apps (Tap Zoo for example) seem to download their state from the server, which comes as a cleartext JSON response. Using your proxy you might be able to substitute your own JSON and change things in interesting ways.

Happy blocking!

pip install objgraph

At the relevant point in your code, add an import pdb; pdb.set_trace() to drop into the debugger. Then just follow the docs on finding memory leaks with objgraph. In short you do:

>> import objgraph
>> objgraph.show_growth(limit=10)   # Start counting

Ignore that output. Call the function that leaks memory, iterate once through you loop, whatever you need to do to make your program consume more memory. Now call show_growth again:

>> my_leaky_func()
>> objgraph.show_growth(limit=10)   # Stop and show change

This time it shows the difference between now and the last time you called it. Those extra objects are the problem.

Finally you need to find where the reference to those leaky objects is being held:

>> import inspect, random
>> objgraph.show_chain(
...     objgraph.find_backref_chain(
...         random.choice(objgraph.by_type('MyBigFatObject')),
...         inspect.ismodule),
...     filename='chain.png')

That generates a lovely graph in your current directory, showing the ownership chain. Now wasn’t that easy? Thanks Marius Gedminas!

For any branch, I want to know whether it has been merged, when the last commit was, and ideally if the matching ticket in our tracker has been closed.

Switch to your main branch, usually develop or master: git checkout develop

List all the branches which have been fully merged into it:

git branch -a --merged

Show last commit date for a branch:

git log -1 --pretty=format:"%Cgreen%ci %Cred%cr%Creset" <branch_name>

Putting all that together, for all branches, gives us (thanks brunost and unixmonkey7740):

for k in $(git branch -a --merged|grep -v "\->"|sed s/^..//);do echo -e $(git log -1 --pretty=format:"%Cgreen%ci %Cred%cr%Creset" "$k")\\t"$k";done|sort|more

Most of my branches are named features/<ticket-id>, so I bring up the above list in one window, and the ticket tracker in a browser, and work through them.

To delete a branch:

# If you have it checked out locally
git branch -d <branch_name>

# Delete remote branch
git push origin :<branch_name>

To keep you safe from typos, git branch -d will refuse to delete branches that haven’t been merged.

Finally, once you have deleted all the old branches, your colleagues will need to bring their remote names in sync:

git remote prune origin

Remember that in git branches are just like symlinks, so you’re not losing any commits by deleting a branch, you’re just losing a named reference to a commit.

Happy pruning!

But what if your app is CPU bound? How do you find out where it’s spending it’s time? It’s easy with the runprofileserver from django-extensions – here’s how:

pip install django-extensions

Add django_extensions to your INSTALLED_APPS, then run the profiling server.

django-admin.py runprofileserver

In your browser, hit the url you’re interested in. runprofileserver will create hotshot files in /tmp/. Ctrl-C out of your server, and fire up ipython.

from hotshot import stats
p = stats.load('/tmp/<your filename here>')

# In order that functions got called
p.sort_stats('cumulative').print_stats(20)

# Slowest method first
p.sort_stats('time').print_stats(20)

Here p is a pstats object. I prefer the cumulative view. Look down it until you see a big drop in the cumulative time – that’s where the time was spent.

Decode all input strings

name = input_name.decode('utf8', 'ignore')

You need to decode all input text: filenames, file contents, console input, database contents, socket data, etc. If you are using Django, it already does this for you, as much as it can.

The trick is figuring out which encoding you have. Here are Python’s supported encodings. The main ones I try are utf_8, latin_1 and cp1250.

Inside, work only with unicode

That means prefixing strings with ‘u’:

my_thing = u'Something'

Be careful when concatenating, always use a u:

my_other = part_one + u'-' + part_two

Encode all output strings

output_name = name.encode('utf8')
print(output_name)

Strings (bytes) have an encoding. The ‘ignore’ parameter to decode tells it to throw away any bytes that aren’t valid UTF8.

Unicode does not have an encoding. Hence you need to decode the byte string, specifying the encoding, to get unicode. When you want to output, you need to encode your unicode back to a byte string, again specifying the encoding.

In Python 3, all strings are Unicode, and happiness fills the land.

Auto packing the repository for optimum performance. You may also
run "git gc" manually. See "git help gc" for more information.
error: cannot lock ref 'HEAD (xyz's conflicted copy 2011-06-02)'
error: cannot lock ref 'refs/heads/master (xyz's conflicted copy 2011-06-02)'
error: failed to run reflog

You just need to delete the offending files from .git/logs/ and run your operation again.