July 21, 2013

Online upgrades in Go

Posted in Software at 05:40 by graham

tl;dr Send your socket fd over a UNIX domain socket: syscall/passfd_test.go.

When your server holds long running connections (WebSocket, long-running HTTP, IRC, XMPP, etc) you often want to be able to upgrade the server without dropping the connections (zero downtime upgrade). In UNIX there are at least two ways to do this:

  1. Inherit the file descriptor
  2. Send the file descriptor over a domain socket

The first one is straightforward, because a UNIX process automatically inherits the file descriptors of it’s parent, except if they have the close-on-exec flag set. Go complicates things a bit by always setting that flag on it’s sockets (in net/sock_cloexec.go). For a child process to inherit it’s parent’s file descriptors, you have to manually add them to ExtraFiles in os/exec/Cmd. There’s an example in TestExtraFiles in os/exec/exec_test.go.

Usually you need to send more that just the connections to the child process. There will be some state, and probably a communication where the child tells the parents it’s ready to take over (after priming it’s cache, for example). Hence the second approach, unix domain sockets, is more interesting.

Here’s a full working example, and how to test it:

  • Build it and run it. It should display it’s pid.
  • Telnet to localhost 1122 from two separate terminals. You should see lots of [v1] Message 1 in your terminals.
  • Now edit the file to change the VERSION number.
  • Rebuild it.
  • Signal it to upgrade with kill -USR2 <pid-displayed-earlier>.
  • The numbers in the telnet sessions should keep increasing, the connections continue, but the version number has changed.

Ta da – an online upgrade!

When reading the code below, skip straight to about line 112 (// We're the parent), as that’s where things start. We’ll go into detail after the code.

The two key calls, for passing the sockets to the child, are:

  1. syscall.UnixRights, which puts socket file descriptors into the ancillay data (also called socket control) part of a message.
  2. syscall.ParseUnixRights which gets socket file descriptors out of the ancillary data part of a message.

We’re not really passing a file descriptor at all, just a reference. The fd on the child process will most likely have a different number than in the parent, but they both point to the same entry in the kernel’s file descriptor table.

Around line 187, when the parent is killed (SIGTERM) it sends it’s connections to the child before exiting. The idea is that the child will start, do any initialization it needs, such as priming it’s cache, and then SIGTERM the parent to signal it’s willingness to take over connections. The SIGTERM is sent by the child line 70.

The domain socket that parent and child use to communicate is created line 148 (syscall.Socketpair). We need to get the read end of that pipe to the child. We use the ExtraFiles method, line 293, to pass it.

We also need the child to know that it’s the child. Line 294 we pass it an environment variables called PARENT_PID. We don’t ever use the value of that environment variable – a --child command line argument would have done just as well.

Finally, and not in the example, the child process may want to wait until the parent is confirmed dead. To check if a process is alive you can send it signal 0:

parent, _ := os.FindProcess(ppid)
err = parent.Signal(syscall.Signal(0))
for err == nil {
    time.Sleep(10 * time.Millisecond)
    // TODO: Don't wait forever
    err = parent.Signal(syscall.Signal(0))
}

If err is nil the signal succeeded, meaning the parent process is still running. See man 2 kill.

Happy restarting!

References

3 Comments »

  1. graham said,

    September 5, 2013 at 04:43

    @daniel: You can’t get the Listener out of an http.Server, but you can pass it in :-) Just call http.Serve(myListener) instead of the usual http.ListenAndServe(..). You create myListener by the usual net.Listen(“tcp”, addr).

  2. daniel said,

    September 3, 2013 at 23:39

    This is very cool. Quick question, how would I adapt this to work with an http.Server? It looks like I can’t get a reference to the connection the server opens. Or is there a way to get a reference easily?

  3. Sławosz said,

    August 15, 2013 at 15:09

    This is simmilar approach to ruby unicorn server, but what about threaded servers?

Leave a Comment

Note: Your comment will only appear on the site once I approve it manually. This can take a day or two. Thanks for taking the time to comment.