June 16, 2021

Rust performance 101 in 5 minutes

Posted in Software at 04:49 by graham

Is your Rust program CPU bound? Here are the very first things you can do on Linux.

First be sure the CPU really is the bottleneck. htop should tell you, or that aircraft-taking-off fan noise.

Second isolate the part you are concerned about into a program (or unit test or benchmark) that you can run repeatably.

Third, get some ballpark measurements by running one of these at least four times:

  • time ./myprog
  • perf stat -e task-clock ./myprog

Build in release mode with link time optimization

Add this to your Cargo.toml:

lto = true
codegen-units = 1

then build it:

cargo build --release

This might be all you need. Release mode makes a huge difference. Then try only one or neither of those two profile.release lines just in case they made things worse. Trust, but verify.

Read the rest of this entry »

June 10, 2021

CPU silently disappointed with your choice of instruction ordering

Posted in Software at 00:10 by graham

An adventure in CPU out-of-order instruction execution.

Pretend this is a distributed systems interview question. Assume x, y, r1 and r2 are all 0.

What are the possible values of r1 and r2?

Thread 1    Thread 2
x = 1      y = 1
r1 = y     r2 = x

I expect you really want to add a lock, an atomic, a memory fence, something! Rust won’t even let you write code like this, so for the Rustaceans assume the variables are all Arc<atomicu8> which we get and set with Ordering::Relaxed. But let’s continue, because I learnt a lot.

What values can r1 and r2 have?

  • Either both threads run at the same time: r1 == 1 && r2 == 1
  • or Thread 1 runs all the way through first: r1 == 0 && r2 == 1
  • or Thread 2 runs all the way through first: r1 == 1 && r2 == 0

What I did not expect BUT IT HAPPENED, is r1 == 0 and r2 == 0

Read the rest of this entry »

January 31, 2021

Traits: Rust’s unifying concept

Posted in Software at 06:49 by graham

Rust’s traits, which come from Haskell’s type classes, are a single concept that unifies all of this:

This is amazing!

Every other language I’ve used [1] has had some subset of the list above. I don’t think any of them have had them all, and they have been distinct concepts with their own keywords and syntax. Being able to unify all of that is frankly wonderful.

Rust doesn’t have a special syntax for interfaces, because traits cover that. Rust doesn’t have a special syntax for operator overloading, because traits cover that. And so on.

To underline how flexible this idea is, here’s how to do everything on the list above with just traits:

Read the rest of this entry »

December 21, 2020

A day in the life of a professional software engineer

Posted in Software at 00:05 by graham

If you’re a graduate interviewing for a software job and wondering what a typical day is like, here’s some notes from what I did last Tuesday. It was a pretty typical day.

First, as soon as I got to work I realized the night shift had set the linked list up backwards. Doh! I quickly went to the whiteboard to figure out how to reverse it. This won’t be a problem much longer as our team is switching to a double-ended linked list soon. We’re presenting to management about it next week.

I grabbed the first task from my work queue: One of our customers has a long string, and they need to find whether a smaller string is contained within it. This is our bread-and-butter as software engineers. I quickly coded up a Rabin-Karp search with a rolling hash, and had the answer for them by 10am.

Coffee, then task two: The QA team rejected the sort implementation I did last week. I’d written a basic Bubble Sort, and QA wanted to beat O(n2) for the average case. No problem, I switched to a Merge Sort and was done by lunchtime.

That reminded me that I hadn’t finished my Big O calculations for the design review later this week, so that was my first task after lunch. Junior engineers only do the ‘n’ part, but as a senior I do the constant as well. I have three more functions to calculate.

Greg, my colleague came over, he needed help balancing his tree. He’s using a Red-Black; those are tricky. After a quick consult with Dave our lead engineer, I helped Greg switch his design to an AVL. He has a lookup-intensive use case, so it’s a better match.

And that’s it, I headed for the bus station and home. But wait, my day wasn’t over! The bus driver’s watch was broken, and he needed to time 15 minutes. Luckily I had with me two ropes of varying density, both of which burn for exactly one hour…

Discuss on reddit

November 19, 2020

Three things I wish I’d known learning Rust

Posted in Software at 01:04 by graham

At Screenleap we’re running Rust in production, yet I feel like I’m still only scratching the surface of the language. Hopefully you will enjoy learning Rust as much as I am.

It will take longer to learn than most languages

As professional programmers we expect to learn a new language every few years, and it usually doesn’t take very long. Learning Rust was different for me for three reasons:

  1. Rust is a big language. The official book is over 500 pages, and it has to skip quite a lot.

  2. It introduces new concepts: Ownership, Lifetimes, Traits. Those aren’t in any other language I’ve used.

  3. It makes explicit things that other languages hide. In other languages you can happily pass values around and change them from anywhere. That language is doing lots of things inside to make it work for you. Rust gives you control over those things. If you have a C++ background this part will be easier for you.

The good news is there are excellent resources for learning Rust, and a very helpful community. The offical Rust Book is the ideal place to start. Rust By Example helped me a lot too.

The standard library is small, you will need dependencies

Learning a language means also learning its standard library, so at first I try to avoid dependencies. That’s not possible in Rust.

There are several semi-official crates that most projects use, and you should pretend those are standard library. Examples include: log, chrono, rand, clap, reqwest and serde.

Once I accepted that the standard library isn’t batteries-included I found it easy to identify the library I needed. The Rust Cookbook is a great place to look, as is the Most Downloaded on crates.io.

A lot of behavior is in Traits

In Rust a lot of methods are not on the object itself, but were added by a trait. In this context think of a trait as a mix-in, a way to add behavior (methods) to an existing object.

This can make it hard to find the method you need. For example, the File object has no obvious methods to either read or write. You have to look under “Trait Implementations”, expand “Read” trait, and finally expand “Show hidden undocumented items”.

Methods will appear on your object when you import a trait. If you have a File struct, you can’t read anything from it until you import the trait that attaches those methods (use std::io::Read). You will sometimes see code call methods on a type (even built in ones like f64) that you can’t call. That’s because they are importing a trait that you aren’t. So use the trait, Luke.

Discuss on reddit /r/rust

June 11, 2020

Quote of the day: A stirring speech

Posted in Behaviour at 01:14 by graham

From Frankenstein; Or, The Modern Prometheus. The north-bound boat is trapped in ice. The crew are demanding the captain abandon the expedition, turn around and go home. Victor Frankenstein leaps up to change their minds:

“What do you mean? What do you demand of your captain? Are you then so easily turned from your design? Did you not call this a glorious expedition? And wherefore was it glorious?

Not because the way was smooth and placid as a southern sea, but because it was full of dangers and terror; because, at every new incident, your fortitude was to be called forth, and your courage exhibited; because danger and death surrounded it, and these you were to brave and overcome. For this was it a glorious, for this was it an honourable undertaking.

You were hereafter to be hailed as the benefactors of your species; your names adored, as belonging to brave men who encountered death for honour, and the benefit of mankind.

And now, behold, with the first imagination of danger, or, if you will, the first mighty and terrific trial of your courage, you shrink away, and are content to be handed down as men who had not strength enough to endure cold and peril; and so, poor souls, they were chilly, and returned to their warm fire-sides. Why, that requires not this preparation; ye need not have come thus far, and dragged your captain to the shame of a defeat, merely to prove yourselves cowards.

Oh! be men, or be more than men. Be steady to your purposes, and firm as a rock. This ice is not made of such stuff as your hearts may be; it is mutable, and cannot withstand you, if you say that it shall not.

Do not return to your families with the stigma of disgrace marked on your brows. Return, as heroes who have fought and conquered, and who know not what it is to turn their backs on the foe.”

Thanks to Mary Shelley and Project Gutenberg.

June 6, 2020

The simplest CLI that could possibly work

Posted in Software at 19:12 by graham

Most teams have many little command-line utilities that need to be kept organised. At Screenleap for example I have written tools in Go to manage build artifacts, releases, servers, and local dev environments, to run test clients in various roles, and many other things.

What’s the simplest possible way to put all your ad-hoc tools in one place? We have a single binary called sl that you run as sl [command] [command options].

The sl main function is basically this:

var commands = map[string]func(){
        "thing1": func1,
        "thing2": pkgA.Main,
func main() {

The commands define their own flags, print errors, everything a normal binary would do. In practice sl should check os.Args[1] and print what commands are available, so here’s the real code:

func main() {
        if len(os.Args) == 1 {
        cmd, ok := commands[os.Args[1]]
        if !ok {

func usage() string {
        s := "Usage: sl [cmd] [options]\nCommands:\n"
        for k := range commands {
                s += " - " + k + "\n"
        return s

Sure you could use one of the Go command line frameworks out there, but then you’re imposing a dependency on your colleagues. We’ve been using the above map-of-functions approach since I joined five years ago because I couldn’t think of a simpler way to start, and so far we’re happy with it.


  • Single binary. Everyone knows where to look for a tool, there’s only one file to `scp` around, and in Go it saves a lot of disk space.
  • Our servers are in Go, and these command line tools use the same libraries as the servers. It makes experimenting with a scenario quite fast, you can script with the server domain objects.
  • No requirements on the individual commands, a command is just a function. The commands print their own usage documentation.
  • No dependencies.

Happy tooling!

May 22, 2020

Collect and handle multiple errors in Go

Posted in Software at 18:33 by graham

In Go, how do you run several operations that might return an error, and return those errors at the end? For example you might be stopping several services:

func stopAll() error {
    if err := stopIndexer(); err != nil {
        // save the error but continue
    if err := stopAuth(); err != nil {
        // save the error but continue
    if err := stopJobs(); err != nil {
        // save the error but continue
    return allTheErrors

There are many ways to do it. Here’s how I do it, maybe it will be useful to you:

func stopAll() error {
    var errs util.Errs // code for Errs is below
    return errs.Ret()

You treat errs.Ret() like any other error. It’s nil if none of the operations returned an error, otherwise it contains the text of all the errors. You can use errors.Is and errors.As on it, it will report if any of the internal errors match.

Why not use one of the many packages other people wrote, or publish this as a package? I try very hard to minimize dependencies. Each dependency imposes a cognitive cost on your colleagues, it is not something that should be done lightly, or alone.

Read the rest of this entry »

September 13, 2019

A developer goes to a DevOps conference

Posted in Software at 18:54 by graham

As a professional software engineer and enthusiastic pro-am systems administrator, I have long been curious about DevOps.

I spent the last two days at PDX DevOpsDays. Across organised talks, Ignite’s, Open Spaces, and as many hallway conversations as I could fit in, here is what I learned about what “DevOps” is and does.

DevOps is Ops with new tools. I went to DevOpsDays thinking DevOps means developers and operations merged into one team. I was wrong.

I did not meet or hear from any developers. It was a gathering of system administrators who use version control, and write a lot of YAML. Programming languages were not a topic; I heard Python mentioned once in passing, that’s it.

DevOps means the veteran admins had to check in their personal scripts, and everyone is expected to automate more things. They relate to the software their business runs (“app”, “asset”) the way a merchant-navy captain relates to what’s in the containers on his ship.

Read the rest of this entry »

July 27, 2018

Reading MediaRecorder’s webm/opus output

Posted in Software at 20:34 by graham

Javascript can now record, manipulate and play sound thanks to a whole range of audio-related API’s. One of the simpler and more useful parts of that is MediaRecorder which can record sound, typically from the users’ microphone.

const stream = await navigator.mediaDevices
    .getUserMedia({audio: true, video: false});  // Media Capture and Streams API
const mediaRecorder = new MediaRecorder(stream); // MediaStream Recording API
mediaRecorder.ondataavailable = (ev) => {        // ev is a BlobEvent
    // ev.data is a Blob. Send it to the server.
mediaRecorder.start(50);  // call ondataavailable every 50ms

MediaRecorder on Chrome produces output with mime type audio/webm;codecs=opus.

Let me guide you through the WebM / Matroska / EBML maze.

The WebM format is a container, which can contain many things. In our case it contains audio encoded with the Opus codec. We are only going to look at the container, not the encoded audio.

Read the rest of this entry »

« Previous entries Next Page » Next Page »