June 11, 2020

Quote of the day: A stirring speech

Posted in Behaviour at 01:14 by graham

From Frankenstein; Or, The Modern Prometheus. The north-bound boat is trapped in ice. The crew are demanding the captain abandon the expedition, turn around and go home. Victor Frankenstein leaps up to change their minds:

“What do you mean? What do you demand of your captain? Are you then so easily turned from your design? Did you not call this a glorious expedition? And wherefore was it glorious?

Not because the way was smooth and placid as a southern sea, but because it was full of dangers and terror; because, at every new incident, your fortitude was to be called forth, and your courage exhibited; because danger and death surrounded it, and these you were to brave and overcome. For this was it a glorious, for this was it an honourable undertaking.

You were hereafter to be hailed as the benefactors of your species; your names adored, as belonging to brave men who encountered death for honour, and the benefit of mankind.

And now, behold, with the first imagination of danger, or, if you will, the first mighty and terrific trial of your courage, you shrink away, and are content to be handed down as men who had not strength enough to endure cold and peril; and so, poor souls, they were chilly, and returned to their warm fire-sides. Why, that requires not this preparation; ye need not have come thus far, and dragged your captain to the shame of a defeat, merely to prove yourselves cowards.

Oh! be men, or be more than men. Be steady to your purposes, and firm as a rock. This ice is not made of such stuff as your hearts may be; it is mutable, and cannot withstand you, if you say that it shall not.

Do not return to your families with the stigma of disgrace marked on your brows. Return, as heroes who have fought and conquered, and who know not what it is to turn their backs on the foe.”

Thanks to Mary Shelley and Project Gutenberg.

June 6, 2020

The simplest CLI that could possibly work

Posted in Software at 19:12 by graham

Most teams have many little command-line utilities that need to be kept organised. At Screenleap for example I have written tools in Go to manage build artifacts, releases, servers, and local dev environments, to run test clients in various roles, and many other things.

What’s the simplest possible way to put all your ad-hoc tools in one place? We have a single binary called sl that you run as sl [command] [command options].

The sl main function is basically this:

var commands = map[string]func(){
        "thing1": func1,
        "thing2": pkgA.Main,
}
func main() {
        commands[os.Args[1]]()
}

The commands define their own flags, print errors, everything a normal binary would do. In practice sl should check os.Args[1] and print what commands are available, so here’s the real code:

func main() {
        if len(os.Args) == 1 {
                fmt.Println(usage())
                os.Exit(1)
        }
        cmd, ok := commands[os.Args[1]]
        if !ok {
                fmt.Println(usage())
                os.Exit(1)
        }
        cmd()
}

func usage() string {
        s := "Usage: sl [cmd] [options]\nCommands:\n"
        for k := range commands {
                s += " - " + k + "\n"
        }
        return s
}

Sure you could use one of the Go command line frameworks out there, but then you’re imposing a dependency on your colleagues. We’ve been using the above map-of-functions approach since I joined five years ago because I couldn’t think of a simpler way to start, and so far we’re happy with it.

Advantages:

  • Single binary. Everyone knows where to look for a tool, there’s only one file to `scp` around, and in Go it saves a lot of disk space.
  • Our servers are in Go, and these command line tools use the same libraries as the servers. It makes experimenting with a scenario quite fast, you can script with the server domain objects.
  • No requirements on the individual commands, a command is just a function. The commands print their own usage documentation.
  • No dependencies.

Happy tooling!

May 22, 2020

Collect and handle multiple errors in Go

Posted in Software at 18:33 by graham

In Go, how do you run several operations that might return an error, and return those errors at the end? For example you might be stopping several services:

func stopAll() error {
    if err := stopIndexer(); err != nil {
        // save the error but continue
    }
    if err := stopAuth(); err != nil {
        // save the error but continue
    }
    if err := stopJobs(); err != nil {
        // save the error but continue
    }
    [...]
    return allTheErrors
}

There are many ways to do it. Here’s how I do it, maybe it will be useful to you:


func stopAll() error {
    var errs util.Errs // code for Errs is below
    errs.Add(stopIndexer())
    errs.Add(stopAuth())
    errs.Add(stopJobs())
    return errs.Ret()
}

You treat errs.Ret() like any other error. It’s nil if none of the operations returned an error, otherwise it contains the text of all the errors. You can use errors.Is and errors.As on it, it will report if any of the internal errors match.

Why not use one of the many packages other people wrote, or publish this as a package? I try very hard to minimize dependencies. Each dependency imposes a cognitive cost on your colleagues, it is not something that should be done lightly, or alone.

Read the rest of this entry »

September 13, 2019

A developer goes to a DevOps conference

Posted in Software at 18:54 by graham

As a professional software engineer and enthusiastic pro-am systems administrator, I have long been curious about DevOps.

I spent the last two days at PDX DevOpsDays. Across organised talks, Ignite’s, Open Spaces, and as many hallway conversations as I could fit in, here is what I learned about what “DevOps” is and does.

DevOps is Ops with new tools. I went to DevOpsDays thinking DevOps means developers and operations merged into one team. I was wrong.

I did not meet or hear from any developers. It was a gathering of system administrators who use version control, and write a lot of YAML. Programming languages were not a topic; I heard Python mentioned once in passing, that’s it.

DevOps means the veteran admins had to check in their personal scripts, and everyone is expected to automate more things. They relate to the software their business runs (“app”, “asset”) the way a merchant-navy captain relates to what’s in the containers on his ship.

Read the rest of this entry »

July 27, 2018

Reading MediaRecorder’s webm/opus output

Posted in Software at 20:34 by graham

Javascript can now record, manipulate and play sound thanks to a whole range of audio-related API’s. One of the simpler and more useful parts of that is MediaRecorder which can record sound, typically from the users’ microphone.

const stream = await navigator.mediaDevices
    .getUserMedia({audio: true, video: false});  // Media Capture and Streams API
const mediaRecorder = new MediaRecorder(stream); // MediaStream Recording API
mediaRecorder.ondataavailable = (ev) => {        // ev is a BlobEvent
    // ev.data is a Blob. Send it to the server.
};
mediaRecorder.start(50);  // call ondataavailable every 50ms

MediaRecorder on Chrome produces output with mime type audio/webm;codecs=opus.

Let me guide you through the WebM / Matroska / EBML maze.

The WebM format is a container, which can contain many things. In our case it contains audio encoded with the Opus codec. We are only going to look at the container, not the encoded audio.

Read the rest of this entry »

Javascript’s async/await and Promise in a few words

Posted in Software at 15:30 by graham

This is a very common pattern in Javascript:

myFunc(value, successCallback, errorCallback) 

If myFunc could block (such as on network access), we don’t want to block Javascript’s single thread, so we asked to be notified when the operation is complete.

Javascript added syntactic sugar to make this pattern more pleasant, in two steps.

The first step is adding a Promise object. myFunc would now returns a Promise, and the caller looks like this:

myFunc(value).then(successCallback, errorCallback)

Very similar. An improvement is you can now return this Promise up the call chain, instead of passing down your callbacks, and you can chain then. I sometimes think of Promise as ‘CallbackManager’. It wraps them and gives them a nicer interface.

Read the rest of this entry »

March 29, 2018

Growing software, in two tweets

Posted in Software at 19:34 by graham

Gardening is my preferred metaphor for software. When I grow a new piece of software, there is a very predictable path I go through as I attempt to manage it’s complexity. It is, in fast-forward, the history of programming language design. That process is short enough to capture in about 400 characters, as two tweets.

Read the rest of this entry »

November 11, 2017

What Made Maddy Run

Posted in Behaviour, Society at 22:58 by graham

What Made Maddy Run by Kate Fagan, is a book about the importance of doing what you love, of really listening to the people close to you. It is a case study of both the valedictorians from Lives of Promise and the dead from Why People Die by Suicide. It could have been a book about Overtraining Syndrome.

It is the factual account of the suicide of Madison Holleran, reconstructed from digital media (email, text, Instagram, etc) with help from her family.

In our safe and well-fed developed world the single biggest killer of teens and adults is suicide. Case studies matter.

Read the rest of this entry »

May 22, 2017

Learn Better – book notes

Posted in Behaviour at 05:25 by graham

Learn Better, by Ulrich Bosner is an interesting, valuable book, that is too long. The information would fit comfortably into 50 pages, but you can’t sell a book that short. Amazon is fixing this, but I have a strong preference for paper books over e-books. Anyway, here are my notes.

The core ideas

There are two big ideas in Learn Better:

I. Learning is a skill which you can improve at

Carol Dweck’s Mindset claims that individuals who believe this (versus thinking the ability to learn is fixed and innate) live “a more successful life”. If you picked up a book called “Learn Better”, I’m guessing you already think you can learn better, so hey, you’re half way there already.

In noticed the same viewpoint in a large study of high-school valedictorians: “The top students readily identified themselves as ‘school smart’. Academic talent, to them, meant the ability to excel at academic learning and school tasks such as note taking, memorization, and testing. Many of them clearly attributed their success primarily to effort rather than ability.”

II. Learning is a generative activity

You don’t learn by loading information from somewhere into your brain. You have to create it. To learn, you must do.

Tangent: I found this interesting because of the similarities with memory. We often think of memory as a video camera, yet recalling a memory is a creative activity. I’d highly recommend Elizabeth Loftus’s The myth of repressed memory which explains this very well.

What does “generative” mean here? What kind of activities must you do to learn?

  • Take notes in your own words as you read (I’m doing it now!).
  • Summarize.
  • Tell other people what you learned (even imaginary other people); this is what we do in programming with the rubber duck.
  • Take practice quizzes.

    Read the rest of this entry »

April 9, 2016

systemd socket activation in Go

Posted in Software at 20:01 by graham

To start a server on a port below 1024 (i.e. 80, 443), you need root permissions or capability CAP_NET_BIND_SERVICE, but you also want most of your server to run unprivileged, reducing your attack surface. The traditional way to achieve this was to start as root, bind the socket, then drop privileges. It’s a hassle, and if you get it wrong it’s a big security hole, so often we just run on an unprivileged port and use something like nginx to proxy. But no longer. There’s a much better way: systemd socket activation.

Here is a Go program that will listen on port 8080 if started manually, or port 80 if run via systemd.

package main

import (
    "log"
    "net"
    "net/http"
    "os"
    "strconv"
)

func main() {
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("Hello World!"))
    })

    if os.Getenv("LISTEN_PID") == strconv.Itoa(os.Getpid()) {
        // systemd run
        f := os.NewFile(3, "from systemd")
        l, err := net.FileListener(f)
        if err != nil {
            log.Fatal(err)
        }
        http.Serve(l, nil)
    } else {
        // manual run
        log.Fatal(http.ListenAndServe(":8080", nil))
    }
}

Build it (go build hello.go) and put the hello binary on your path (~/bin/ in my case).

Create /etc/systemd/system/hello.service:

[Unit]
Requires=hello.socket

[Service]
ExecStart=/home/graham/bin/hello
NonBlocking=true

Create /etc/systemd/system/hello.socket:

[Unit]
Description=Hello socket

[Socket]
ListenStream=80
NoDelay=true

We use NoDelay (TCP_NODELAY) and NonBlocking (O_NONBLOCK) because Go sets these by default on sockets it opens, and we want similar behavior.

Reload the config in systemd (sudo systemctl daemon-reload), and you’re done. If you start it manually it will listen on 8080, if you start it via sudo systemctl start hello it will listen on port 80.

To stop the service you also need to stop the socket. If you don’t stop the socket, incoming connections will auto-start the service (inetd style). systemd will warn you about this.

sudo systemctl stop hello.socket hello

How it works

When systemd starts a process that uses socket-based activation it sets the following environment variables. To check whether we are being started via socket activation we just need to check if one of those environment variables is set.

  • LISTEN_PID: The process id of the process who gets the sockets. This prevents a child forked from your main process from thinking it is being given some sockets.
  • LISTEN_FDS: The number of file descriptors (sockets) your process is being given. In our case this will be 1.

Every process gets three standard file descriptors: STDIN=0, STDOUT=1, and STDERR=2. Descriptors given to us by systemd hence start at number 3.

Here are all the .socket file options. You should also set protections in your .service file, see The joy of systemd.

TLS and HTTP2

Edit /etc/systemd/system/hello.socket to use port 443:

ListenStream=443

Wrap the listener with tls.NewListener, and set tls.Config.NextProtos to be []string{"h2", "http/1.1"} to maintain http2 support. Here’s the full // systemd run section:

config := &tls.Config{
    Certificates:             make([]tls.Certificate, 1),
    NextProtos:               []string{"h2", "http/1.1"},
    PreferServerCipherSuites: true,
}
var err error
config.Certificates[0], err = tls.LoadX509KeyPair(
    "my_cert.pem", 
    "my_cert.key",
)
if err != nil {
    log.Fatal(err)
}
f := os.NewFile(3, "from systemd")
l, err := net.FileListener(f)
if err != nil {
    log.Fatal(err)
}
tlsListener := tls.NewListener(l, config)
http.Serve(tlsListener, nil)

That’s it!

UPDATE 2018: Vincent Bernat has an excellent blog-post on zero-downtime upgrades in Go with systemd.

« Previous entries Next Page » Next Page »