June 6, 2020

The simplest CLI that could possibly work

Posted in Software at 19:12 by graham

Most teams have many little command-line utilities that need to be kept organised. At Screenleap for example I have written tools in Go to manage build artifacts, releases, servers, and local dev environments, to run test clients in various roles, and many other things.

What’s the simplest possible way to put all your ad-hoc tools in one place? We have a single binary called sl that you run as sl [command] [command options].

The sl main function is basically this:

var commands = map[string]func(){
        "thing1": func1,
        "thing2": pkgA.Main,
func main() {

The commands define their own flags, print errors, everything a normal binary would do. In practice sl should check os.Args[1] and print what commands are available, so here’s the real code:

func main() {
        if len(os.Args) == 1 {
        cmd, ok := commands[os.Args[1]]
        if !ok {

func usage() string {
        s := "Usage: sl [cmd] [options]\nCommands:\n"
        for k := range commands {
                s += " - " + k + "\n"
        return s

Sure you could use one of the Go command line frameworks out there, but then you’re imposing a dependency on your colleagues. We’ve been using the above map-of-functions approach since I joined five years ago because I couldn’t think of a simpler way to start, and so far we’re happy with it.


  • Single binary. Everyone knows where to look for a tool, there’s only one file to `scp` around, and in Go it saves a lot of disk space.
  • Our servers are in Go, and these command line tools use the same libraries as the servers. It makes experimenting with a scenario quite fast, you can script with the server domain objects.
  • No requirements on the individual commands, a command is just a function. The commands print their own usage documentation.
  • No dependencies.

Happy tooling!

May 22, 2020

Collect and handle multiple errors in Go

Posted in Software at 18:33 by graham

In Go, how do you run several operations that might return an error, and return those errors at the end? For example you might be stopping several services:

func stopAll() error {
    if err := stopIndexer(); err != nil {
        // save the error but continue
    if err := stopAuth(); err != nil {
        // save the error but continue
    if err := stopJobs(); err != nil {
        // save the error but continue
    return allTheErrors

There are many ways to do it. Here’s how I do it, maybe it will be useful to you:

func stopAll() error {
    var errs util.Errs // code for Errs is below
    return errs.Ret()

You treat errs.Ret() like any other error. It’s nil if none of the operations returned an error, otherwise it contains the text of all the errors. You can use errors.Is and errors.As on it, it will report if any of the internal errors match.

Why not use one of the many packages other people wrote, or publish this as a package? I try very hard to minimize dependencies. Each dependency imposes a cognitive cost on your colleagues, it is not something that should be done lightly, or alone.

Read the rest of this entry »

September 13, 2019

A developer goes to a DevOps conference

Posted in Software at 18:54 by graham

As a professional software engineer and enthusiastic pro-am systems administrator, I have long been curious about DevOps.

I spent the last two days at PDX DevOpsDays. Across organised talks, Ignite’s, Open Spaces, and as many hallway conversations as I could fit in, here is what I learned about what “DevOps” is and does.

DevOps is Ops with new tools. I went to DevOpsDays thinking DevOps means developers and operations merged into one team. I was wrong.

I did not meet or hear from any developers. It was a gathering of system administrators who use version control, and write a lot of YAML. Programming languages were not a topic; I heard Python mentioned once in passing, that’s it.

DevOps means the veteran admins had to check in their personal scripts, and everyone is expected to automate more things. They relate to the software their business runs (“app”, “asset”) the way a merchant-navy captain relates to what’s in the containers on his ship.

Read the rest of this entry »

July 27, 2018

Reading MediaRecorder’s webm/opus output

Posted in Software at 20:34 by graham

Javascript can now record, manipulate and play sound thanks to a whole range of audio-related API’s. One of the simpler and more useful parts of that is MediaRecorder which can record sound, typically from the users’ microphone.

const stream = await navigator.mediaDevices
    .getUserMedia({audio: true, video: false});  // Media Capture and Streams API
const mediaRecorder = new MediaRecorder(stream); // MediaStream Recording API
mediaRecorder.ondataavailable = (ev) => {        // ev is a BlobEvent
    // ev.data is a Blob. Send it to the server.
mediaRecorder.start(50);  // call ondataavailable every 50ms

MediaRecorder on Chrome produces output with mime type audio/webm;codecs=opus.

Let me guide you through the WebM / Matroska / EBML maze.

The WebM format is a container, which can contain many things. In our case it contains audio encoded with the Opus codec. We are only going to look at the container, not the encoded audio.

Read the rest of this entry »

Javascript’s async/await and Promise in a few words

Posted in Software at 15:30 by graham

This is a very common pattern in Javascript:

myFunc(value, successCallback, errorCallback) 

If myFunc could block (such as on network access), we don’t want to block Javascript’s single thread, so we asked to be notified when the operation is complete.

Javascript added syntactic sugar to make this pattern more pleasant, in two steps.

The first step is adding a Promise object. myFunc would now returns a Promise, and the caller looks like this:

myFunc(value).then(successCallback, errorCallback)

Very similar. An improvement is you can now return this Promise up the call chain, instead of passing down your callbacks, and you can chain then. I sometimes think of Promise as ‘CallbackManager’. It wraps them and gives them a nicer interface.

Read the rest of this entry »

March 29, 2018

Growing software, in two tweets

Posted in Software at 19:34 by graham

Gardening is my preferred metaphor for software. When I grow a new piece of software, there is a very predictable path I go through as I attempt to manage it’s complexity. It is, in fast-forward, the history of programming language design. That process is short enough to capture in about 400 characters, as two tweets.

Read the rest of this entry »

April 9, 2016

systemd socket activation in Go

Posted in Software at 20:01 by graham

To start a server on a port below 1024 (i.e. 80, 443), you need root permissions or capability CAP_NET_BIND_SERVICE, but you also want most of your server to run unprivileged, reducing your attack surface. The traditional way to achieve this was to start as root, bind the socket, then drop privileges. It’s a hassle, and if you get it wrong it’s a big security hole, so often we just run on an unprivileged port and use something like nginx to proxy. But no longer. There’s a much better way: systemd socket activation.

Here is a Go program that will listen on port 8080 if started manually, or port 80 if run via systemd.

package main

import (

func main() {
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("Hello World!"))

    if os.Getenv("LISTEN_PID") == strconv.Itoa(os.Getpid()) {
        // systemd run
        f := os.NewFile(3, "from systemd")
        l, err := net.FileListener(f)
        if err != nil {
        http.Serve(l, nil)
    } else {
        // manual run
        log.Fatal(http.ListenAndServe(":8080", nil))

Build it (go build hello.go) and put the hello binary on your path (~/bin/ in my case).

Create /etc/systemd/system/hello.service:



Create /etc/systemd/system/hello.socket:

Description=Hello socket


We use NoDelay (TCP_NODELAY) and NonBlocking (O_NONBLOCK) because Go sets these by default on sockets it opens, and we want similar behavior.

Reload the config in systemd (sudo systemctl daemon-reload), and you’re done. If you start it manually it will listen on 8080, if you start it via sudo systemctl start hello it will listen on port 80.

To stop the service you also need to stop the socket. If you don’t stop the socket, incoming connections will auto-start the service (inetd style). systemd will warn you about this.

sudo systemctl stop hello.socket hello

How it works

When systemd starts a process that uses socket-based activation it sets the following environment variables. To check whether we are being started via socket activation we just need to check if one of those environment variables is set.

  • LISTEN_PID: The process id of the process who gets the sockets. This prevents a child forked from your main process from thinking it is being given some sockets.
  • LISTEN_FDS: The number of file descriptors (sockets) your process is being given. In our case this will be 1.

Every process gets three standard file descriptors: STDIN=0, STDOUT=1, and STDERR=2. Descriptors given to us by systemd hence start at number 3.

Here are all the .socket file options. You should also set protections in your .service file, see The joy of systemd.


Edit /etc/systemd/system/hello.socket to use port 443:


Wrap the listener with tls.NewListener, and set tls.Config.NextProtos to be []string{"h2", "http/1.1"} to maintain http2 support. Here’s the full // systemd run section:

config := &tls.Config{
    Certificates:             make([]tls.Certificate, 1),
    NextProtos:               []string{"h2", "http/1.1"},
    PreferServerCipherSuites: true,
var err error
config.Certificates[0], err = tls.LoadX509KeyPair(
if err != nil {
f := os.NewFile(3, "from systemd")
l, err := net.FileListener(f)
if err != nil {
tlsListener := tls.NewListener(l, config)
http.Serve(tlsListener, nil)

That’s it!

UPDATE 2018: Vincent Bernat has an excellent blog-post on zero-downtime upgrades in Go with systemd.

January 12, 2016

The Joy of systemd

Posted in Software at 17:51 by graham

Three years ago when I wrote The Joy of Upstart, that was the easiest way to turn your scripts into daemons. Today the future belongs firmly to systemd, so let’s revisit the Upstart post, and update it for systemd.

systemd is here, it is the default on most major distributions including Ubuntu from 15.04, Fedora from 15, and Red Hat from v7, so you can probably use it now. The main exception is Ubuntu LTS, where we need to wait a few more months for 16.04.

Take this python script: /home/myuser/ez_daemon.py

import time
while 1:
    print("I'm a daemon!")

We’re going to turn it into a daemon in just two lines. Create /etc/systemd/system/ez_daemon.service:

ExecStart=/usr/bin/python -u /home/myuser/ez_daemon.py

And that’s it!

  • systemctl start ez_daemon
  • journalctl -f -u ez_daemon. Stdout goes to the journal by default (more below).
  • systemctl status ez_daemon. The status page is actually useful.
  • systemctl stop ez_daemon

A systemd file (such as /etc/systemd/system/ez_daemon.service) is called a unit. Paths in units must be absolute, hence /usr/bin/python. systemd caches units. When you change one you must systemctl daemon-reload to use the latest version. systemd will remind you.

A major difference with Upstart is that stdout and stderr are now buffered. Upstart went the extra mile by wrapping our script with a pseudo-tty to prevent the buffering. systemd uses a pipe, which means by default there is a 4k buffer on stdout and stderr. You won’t see anything in journalctl until that buffer fills. We pass -u to python to prevent this buffering, or we could have called sys.stdout.flush() after each print. Note also that you can’t have end-of-line comments, a comment must be on a line by itself.

Here’s a more complete example:

Description="Example daemon"
# Don't start until the network is available

ExecStart=/usr/bin/python -u /home/myuser/bin/ez_daemon.py
    PYTHONPATH=/srv/example/src/example/ \
    ANSWER=42 \
    "GREET=Hello systemd"
# /usr, /boot and /etc are read-only
ProtectSystem=full    # even safer: ProtectSystem=strict
# $HOME is read only ..
# .. except /home/myuser/logs
# /tmp is isolated from all other processes
# Minimal /dev, no physical device access
# Don't allow process to raise privileges (e.g. disable suid)
# No network access


A particularly exciting part of systemd is the security features enabled by cgroups. The example script includes most of them, which you’ll want to remove as necessary. For example PrivateNetwork wouldn’t be applicable for nginx. Here are full details on systemd security. UPDATE 2018: systemd now has the even safer ProtectSystem=strict and dynamic users.

systemd replaces syslog. As a convenience Ubuntu still runs syslog, messages are available in /var/log/syslog. Fedora no longer runs syslog by default, so /var/log/messages is empty. Instead you use journalctl, which has greatly improving filtering capabilities. Examples in Linux Voice.

Another exciting feature of systemd is socket activation, for example systemd can open port 80 and hand it to your process. You no longer need to start as root, open the port, then drop privileges. This USENIX Login has examples, and is probably the best place to continue reading after this post.

Ubuntu maintains tips for converting Upstart scripts to systemd.

systemd can also replace cron, run containers, and much more. It’s our future, and it’s going to be amazing.

November 1, 2015

Facebook’s code quality problem

Posted in Software at 00:15 by graham

tl;dr: It looks like Facebook is getting the textbook results of ignoring code quality.

Update: More examples, and insights from ex-employees in the reddit discussion

Facebook has a software quality problem. I’m going to try to convince you with three examples. This is important because it demonstrates the time-honored principle that quality matters. In demonstrates it, as Facebook engineers like to say, at scale. I don’t work at Facebook or any competitor, I’m just an observer.

Exhibit A: “iOS can’t handle our scale”

About a month ago a Facebook engineer gave this presentation: iOS at Facebook, which was followed by a discussion on reddit.

The Facebook iOS app has over 18,000 Objective-C classes, and in a single week 429 people contributing to it. That’s 429 people working, in some way, on the Facebook iOS app. Rather than take the obvious lesson that there are too many people working on this application, the presentation goes on to blame everything from git to Xcode for those 18,000 classes.

Read the rest of this entry »

October 26, 2015

Decrypt your HTTPS traffic with mitmproxy

Posted in Software at 01:18 by graham

tl;dr Use the mitmproxy doc, return here if trouble.

I am porting a server from Java to Go, and need to watch the traffic it receives. The clients include OSX and Windows desktop apps, talking to the server over HTTPS. Here’s how I did that with mitmproxy and iptables. We will setup a transparent proxy.

Instructions are for Ubuntu (arguably the most popular server operating system), but mitmproxy will also run on OSX (anecdotaly the most popular developer laptop) using pf instead of iptables.

Read the rest of this entry »

« Previous entries Next Page » Next Page »