Welcome ! This is the personal site / blog of Graham King. Most people come for the credit card generator, but I think the Categories (top right) are more interesting.

April 9, 2016

systemd socket activation in Go

Posted in Software at 20:01 by graham

To start a server on a port below 1024 (i.e. 80, 443), you need root permissions or capability CAP_NET_BIND_SERVICE, but you also want most of your server to run unprivileged, reducing your attack surface. The traditional way to achieve this was to start as root, bind the socket, then drop privileges. It’s a hassle, and if you get it wrong it’s a big security hole, so often we just run on an unprivileged port and use something like nginx to proxy. But no longer. There’s a much better way: systemd socket activation.

Here is a Go program that will listen on port 8080 if started manually, or port 80 if run via systemd.

package main

import (
    "log"
    "net"
    "net/http"
    "os"
    "strconv"
)

func main() {
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("Hello World!"))
    })

    if os.Getenv("LISTEN_PID") == strconv.Itoa(os.Getpid()) {
        // systemd run
        f := os.NewFile(3, "from systemd")
        l, err := net.FileListener(f)
        if err != nil {
            log.Fatal(err)
        }
        http.Serve(l, nil)
    } else {
        // manual run
        log.Fatal(http.ListenAndServe(":8080", nil))
    }
}

Build it (go build hello.go) and put the hello binary on your path (~/bin/ in my case).

Create /etc/systemd/system/hello.service:

[Unit]
Requires=hello.socket

[Service]
ExecStart=/home/graham/bin/hello
NonBlocking=true

Create /etc/systemd/system/hello.socket:

[Unit]
Description=Hello socket

[Socket]
ListenStream=80
NoDelay=true

We use NoDelay (TCP_NODELAY) and NonBlocking (O_NONBLOCK) because Go sets these by default on sockets it opens, and we want similar behavior.

Reload the config in systemd (sudo systemctl daemon-reload), and you’re done. If you start it manually it will listen on 8080, if you start it via sudo systemctl start hello it will listen on port 80.

To stop the service you also need to stop the socket. If you don’t stop the socket, incoming connections will auto-start the service (inetd style). systemd will warn you about this.

sudo systemctl stop hello.socket hello

How it works

When systemd starts a process that uses socket-based activation it sets the following environment variables. To check whether we are being started via socket activation we just need to check if one of those environment variables is set.

  • LISTEN_PID: The process id of the process who gets the sockets. This prevents a child forked from your main process from thinking it is being given some sockets.
  • LISTEN_FDS: The number of file descriptors (sockets) your process is being given. In our case this will be 1.

Every process gets three standard file descriptors: STDIN=0, STDOUT=1, and STDERR=2. Descriptors given to us by systemd hence start at number 3.

Here are all the .socket file options. You should also set protections in your .service file, see The joy of systemd.

TLS and HTTP2

Edit /etc/systemd/system/hello.socket to use port 443:

ListenStream=443

Wrap the listener with tls.NewListener, and set tls.Config.NextProtos to be []string{"h2", "http/1.1"} to maintain http2 support. Here’s the full // systemd run section:

config := &tls.Config{
    Certificates:             make([]tls.Certificate, 1),
    NextProtos:               []string{"h2", "http/1.1"},
    PreferServerCipherSuites: true,
}
var err error
config.Certificates[0], err = tls.LoadX509KeyPair(
    "my_cert.pem", 
    "my_cert.key",
)
if err != nil {
    log.Fatal(err)
}
f := os.NewFile(3, "from systemd")
l, err := net.FileListener(f)
if err != nil {
    log.Fatal(err)
}
tlsListener := tls.NewListener(l, config)
http.Serve(tlsListener, nil)

That’s it!

January 12, 2016

The Joy of systemd

Posted in Software at 17:51 by graham

Three years ago when I wrote The Joy of Upstart, that was the easiest way to turn your scripts into daemons. Today the future belongs firmly to systemd, so let’s revisit the Upstart post, and update it for systemd.

systemd is here, it is the default on most major distributions including Ubuntu from 15.04, Fedora from 15, and Red Hat from v7, so you can probably use it now. The main exception is Ubuntu LTS, where we need to wait a few more months for 16.04.

Take this python script: /home/myuser/ez_daemon.py

import time
while 1:
    print("I'm a daemon!")
time.sleep(1)

We’re going to turn it into a daemon in just two lines. Create /etc/systemd/system/ez_daemon.service:

[Service]
ExecStart=/usr/bin/python -u /home/myuser/ez_daemon.py

And that’s it!

  • systemctl start ez_daemon
  • journalctl -f -u ez_daemon. Stdout goes to the journal by default (more below).
  • systemctl status ez_daemon. The status page is actually useful.
  • systemctl stop ez_daemon

A systemd file (such as /etc/systemd/system/ez_daemon.service) is called a unit. Paths in units must be absolute, hence /usr/bin/python. systemd caches units. When you change one you must systemctl daemon-reload to use the latest version. systemd will remind you.

A major difference with Upstart is that stdout and stderr are now buffered. Upstart went the extra mile by wrapping our script with a pseudo-tty to prevent the buffering. systemd uses a pipe, which means by default there is a 4k buffer on stdout and stderr. You won’t see anything in journalctl until that buffer fills. We pass -u to python to prevent this buffering, or we could have called sys.stdout.flush() after each print. Note also that you can’t have end-of-line comments, a comment must be on a line by itself.

Here’s a more complete example:

[Unit]
Description="Example daemon"
# Don't start until the network is available
Requires=network-online.target
After=network-online.target

[Service]
ExecStart=/usr/bin/python -u /home/myuser/bin/ez_daemon.py
Restart=on-failure
User=www-data
Group=www-data
Environment=\
    PYTHONPATH=/srv/example/src/example/ \
    ANSWER=42 \
    "GREET=Hello systemd"
# /usr, /boot and /etc are read-only
ProtectSystem=full
# $HOME is read only ..
ProtectHome=read-only
# .. except /home/myuser/logs
ReadWriteDirectories=/home/myuser/logs
# /tmp is isolated from all other processes
PrivateTmp=true
# Minimal /dev, no physical device access
PrivateDevices=true 
# Don't allow process to raise privileges (e.g. disable suid)
NoNewPrivileges=true
# No network access
PrivateNetwork=true

[Install]
WantedBy=multi-user.target

A particularly exciting part of systemd is the security features enabled by cgroups. The example script includes most of them, which you’ll want to remove as necessary. For example PrivateNetwork wouldn’t be applicable for nginx. Here are full details on systemd security.

systemd replaces syslog. As a convenience Ubuntu still runs syslog, messages are available in /var/log/syslog. Fedora no longer runs syslog by default, so /var/log/messages is empty. Instead you use journalctl, which has greatly improving filtering capabilities. Examples in Linux Voice.

Another exciting feature of systemd is socket activation, for example systemd can open port 80 and hand it to your process. You no longer need to start as root, open the port, then drop privileges. This USENIX Login has examples, and is probably the best place to continue reading after this post.

Ubuntu maintains tips for converting Upstart scripts to systemd.

systemd can also replace cron, run containers, and much more. It’s our future, and it’s going to be amazing.

December 11, 2015

How Ada Lovelace solved problems

Posted in Ideas at 06:02 by graham

Over 170 years ago, on Friday 21st July 1843, at 4 o’clock, Ada Lovelace was working on a mathematics problem, possibly on the first known computer program (it was written that summer).

Specifically she was writing extensive notes on her translation of a paper about Charles Babbage’s Analytical Engine, and she was collaborating closely with Babbage to do so. Had it been built the Analytical Engine would have been the first universal computer. Those notes include the first known computer program (and trace of it’s manual execution), which calculates Bernoulli numbers.

But at 4pm on Friday 21st July, she was stuck. It just wasn’t working. We know because she wrote this letter to Babbage:

My dear Babbage. I am in much dismay at having got into so amazing a quagmire and botheration with these numbers, that I cannot possibly get the thing done today. I have no doubts it will all come out clean enough tomorrow; & I shall send you a parcel up, as early in the day as I can. So do no be uneasy. (Tho’ at this moment I am in a charming state of confusion; but it is that sort of confusion which is of a very bubble nature).

I am now going out on horseback. Tant mieux.

Yours puzzle-pate

That ever happened to you? It’s 4pm on a Friday, and it just isn’t working? You know it will be obvious in the morning. So what did Ada do? She went for a horseback ride!

The tagline of this blog is Solvitas Perambulum, Latin for “solve it as you walk”. I find it particularly charming that 172 years ago, the first computer programmer solved it by going for a horseback ride.

Pictures of the original letter can be found in Stephen Wolfram’s fantastic blog post: Untangling the tale of Ada Lovelace.

Happy walking.

November 1, 2015

Facebook’s code quality problem

Posted in Software at 00:15 by graham

tl;dr: It looks like Facebook is getting the textbook results of ignoring code quality.

Update: More examples, and insights from ex-employees in the reddit discussion

Facebook has a software quality problem. I’m going to try to convince you with three examples. This is important because it demonstrates the time-honored principle that quality matters. In demonstrates it, as Facebook engineers like to say, at scale. I don’t work at Facebook or any competitor, I’m just an observer.

Exhibit A: “iOS can’t handle our scale”

About a month ago a Facebook engineer gave this presentation: iOS at Facebook, which was followed by a discussion on reddit.

The Facebook iOS app has over 18,000 Objective-C classes, and in a single week 429 people contributing to it. That’s 429 people working, in some way, on the Facebook iOS app. Rather than take the obvious lesson that there are too many people working on this application, the presentation goes on to blame everything from git to Xcode for those 18,000 classes.

Read the rest of this entry »

October 26, 2015

Decrypt your HTTPS traffic with mitmproxy

Posted in Software at 01:18 by graham

tl;dr Use the mitmproxy doc, return here if trouble.

I am porting a server from Java to Go, and need to watch the traffic it receives. The clients include OSX and Windows desktop apps, talking to the server over HTTPS. Here’s how I did that with mitmproxy and iptables. We will setup a transparent proxy.

Instructions are for Ubuntu (arguably the most popular server operating system), but mitmproxy will also run on OSX (anecdotaly the most popular developer laptop) using pf instead of iptables.

Read the rest of this entry »

August 26, 2015

OSCON 2015: Impressions

Posted in Software at 04:19 by graham

I attended OSCON for the first time this year in Portland. This was a dream come true for me, I have wanted to attend for many years. Here are my impressions and some talk links. I can’t link you to most of the talks, because O’Reilly sells them for $1,000. Open Source indeed.

First, it is a very big conference. They had 1,200 talk proposals, 4,700 attendees, up to 13 talks running concurrently, and two competing mobile apps. Twice I walked into a random Portland bar, which turned out to be hosting an OSCON-related event. It’s that big.

Second, it is a strange mixture of open-source and commercialism. Sponsors above a certain level get a keynote. Tremendous efforts are expended to herd the attendees into the expo hall at every opportunity, towards the paying booths. And I can’t link you to the best talks, because as mentioned O’Reilly sells them. On the other hand all sorts of people I admire in the tech world were there, and I had some wonderful conversations with them, and at the booths in the open-source section. At OSCON, you get to be a peer of the people whose blogs and books you read, and that’s really special.

Here are the speaker slides, a youtube search and, of the public ones, my favorites:

See you next year, in Austin!

July 31, 2015

GopherCon 2015: Favorite talks

Posted in Software at 04:58 by graham

GopherCon was in Denver again this year, and a lot of fun it was, mostly the meeting of wonderful people. Judging by the sponsors, Go continues to be the language of infrastructure (docker, coreos, etcd, mesos, kubernetes, influxdb, etc). Here are my three favorite talks:

  • Rick Hudson: Go GC: Latency problem solved. This was probably the highlight of the conference for me. Rick is a fun speaker, and he’s talking about the key feature in Go 1.5. The garbage collector used to freeze your application for increasingly large amounts of time as you used more memory. It no longer does.
  • Dmitry Vyukov: Go Dynamic Tools. Dmitry covers three tools: The race detector, go-fuzz, and the new execution tracer. The Go tooling is amazing.
  • Audrey Lim: A Beginner’s Mind. Audrey is a lawyer who became a programmer recently, and she recounts her journey – but wait, read on. I confess when her talk was announced I thought I would not find it interesting. I was very wrong, because Audrey is phenomenal. She tried Python, Ruby and Node JS first, and detailed exactly why those were hard for a complete beginner. She made some very astute observations about how the language you use shapes how you think about programming, and the projects you will undertake.

Here are all the other GopherCon 2015 videos.

Robert Griesemer’s keynote highlighted the Pascal (via Oberon) heritage of Go, and remarked that Pascal is a European language family. The C heritage of Go that is more commonly referenced is North American. He hoped Go is a merging of the two software cultures.

And finally, joke of the conference has to go to Tomás Senart.

July 30, 2015

We are Equality

Posted in Society at 05:33 by graham

When the President of the United States of America wants to send an email, we don’t close email-space, and delay your email so his very important one can go through. A homeless person in a public library has exactly the same email service as the richest, most powerful person you can imagine. They both use services such as gmail or yahoo, and beneath that are the same SMTP servers. There is no combination of money, power, or personal connections that can secure you a “better” email service. In email, we are equal.

Pregnant Nigerian teenagers and the ruling family of Saudi Arabia use Facebook or Twitter at the exact same service level, with the same user interface. If you want to write a document, you can use Google Docs, which is what Google’s internal teams use during a crisis. Whoever you are, you can use the same tools, for free, as the people who build the tools.

You can raise money on the same platforms as celebrity musicians or basketball players (Kickstarter, Indiegogo). The Encyclopedia Britannica used to cost upwards of $1,500. Today, Wikipedia provides far more knowledge, for free, for everyone, everywhere. You can rent as many powerful computers as you want, for $1/hour (Linode, Digital Ocean). As the joke goes, on the Internet, no-ones knows you’re a dog.

Technology, and particularly software, have been an incredible equalizing force.

If you have $200 for a computer (or much less for a used model), and occasional access to both the Internet and electricity, you can use, for free, the same tools that a 20-year computer industry veteran such as myself uses (Ubuntu Linux, Go). You have access to all the same learning materials as I do. You can participate in the same forums, communicate with the same people, on the same terms. In the words of the Hacker Manifesto: “If it makes a mistake, it’s because I screwed it up. Not because it doesn’t like me…”.

When you stop to think about it, this is unprecedented. Powerful, affluent, in-group people have always had better homes, transport, and food, breathed better air, and lived longer lives, than the less powerful. There is almost no other facet of human existence where we are all treated exactly equally. It is debatable whether humans, in the physical world, are even capable of this.

All this is possible, maybe inevitable, because of the tools that we in the software industry have built, because of the Internet, and the Open Source model. It makes me incredibly proud to call myself your equal.

July 24, 2015

Building shared libraries in Go: Part 2

Posted in Software at 22:12 by graham

In part 1 we called a very simple Go shared library from Python. Let’s do a more complex example, passing string and []byte, from C++, and getting back a []byte.

Calling Go from C++

Save the following as concat/main.go:

package main

import "C"

//export Concat
func Concat(sIn string, bIn []byte, bOut []byte) {
    n := copy(bOut, sIn)
    copy(bOut[n:], bIn)
}

func main() {}

We add the import "C" so that cgo gives us a header file to #include. Build the shared library and header:

go build -buildmode=c-shared -o libconcat.so concat

In libconcat.h we have this signature:

extern void Concat(GoString p0, GoSlice p1, GoSlice p2)

GoString and GoSlice are defined further up in the header file like this:

typedef struct { char *p; GoInt n; } GoString;
typedef struct { void *data; GoInt len; GoInt cap; } GoSlice;

Copy libconcat.so to /usr/lib/ (or wherever your libraries live). Now let’s call Concat from C++:

#include <vector>
#include <string>
#include <iostream>
#include "libconcat.h"

int main() {
    std::string s_in {"Hello "};
    std::vector<char> v_in {'W', 'o', 'r', 'l', 'd'};
    std::vector<char> v_out(11);

    GoString go_s_in{&s_in[0], static_cast<GoInt>(s_in.size())};
    GoSlice go_v_in{
        v_in.data(),
        static_cast<GoInt>(v_in.size()),
        static_cast<GoInt>(v_in.size()),
    };
    GoSlice go_v_out{
        v_out.data(),
        static_cast<GoInt>(v_out.size()),
        static_cast<GoInt>(v_out.size()),
    };

    Concat(go_s_in, go_v_in, go_v_out);

    for(auto& c : v_out) {
        std::cout << c;
    }
    std::cout << '\n';
}

Save that as concat.cpp. Copy libconcat.h into the same directory. Build and run:

g++ --std=c++14 concat.cpp -o concat -lconcat
./concat

I need the static_cast<GoInt> because a GoInt is a signed type (long long on my machine), but size() returns unsigned type size_t. Apart from wrapping things in Go[String|Slice|etc], this is exactly like calling an ordinary shared libary, because that’s what we built, an ordinary shared library.

The one very important caveat is that you should not pass pointers to Go allocated memory back to the C++ side; that’s why we use an output parameter (v_out). Even if you maintain a reference to it on the Go side so that the garbage collector doesn’t reclaim it, the Go runtime reserves the right to move that memory if they build a copying garbage collector. Lots of details in issue #8310.

Passing C++ allocated memory to Go is fine. Go will not garbage collect memory it did not allocate. There rules here are the same as for cgo.

July 15, 2015

Building shared libraries in Go: Part 1

Posted in Software at 03:22 by graham

Since 1.5, we can write C-style shared libraries (.so) in Go. These can be used directly from C and C++, but also from any language that can use a C shared library, which is most languages. Here’s the original design document. Let’s do a very simple example and call it from Python. In Part 2 we’ll do a more complex example and call it from C++.

Calling Go from Python

package main

import "C"

//export DoubleIt
func DoubleIt(x int) int {
        return x * 2
}

func main() {}

Save that in your GOPATH as doubler/main.go. The exported symbols (the functions the shared library provides) must be in a main package, with a required but ignored main function. They must be marked with magic comment //export <name> (no space), because go build will call cgo.

go build -o libdoubler.so -buildmode=c-shared doubler

This puts libdoubler.so in your current directory. Let’s check we really have a dynamic shared library:

$ file libdoubler.so 
libdoubler.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked ...

$ nm -D libdoubler.so | grep "T DoubleIt"
000000000005d230 T DoubleIt

Well that looks good. Let’s call it from Python, because Python’s ctypes makes this incredibly easy:

>>> import ctypes
>>> lib = ctypes.CDLL("libdoubler.so")
>>> lib.DoubleIt(21)
42

Ta da!

For me this ability to build normal shared libraries makes Go a first-class language on Unix. In later posts we’ll do a more complex example with C++, and time permitting a SWIG example with Ruby.

« Previous entries Next Page » Next Page »