July 30, 2015

We are Equality

Posted in Society at 05:33 by graham

When the President of the United States of America wants to send an email, we don’t close email-space, and delay your email so his very important one can go through. A homeless person in a public library has exactly the same email service as the richest, most powerful person you can imagine. They both use services such as gmail or yahoo, and beneath that are the same SMTP servers. There is no combination of money, power, or personal connections that can secure you a “better” email service. In email, we are equal.

Pregnant Nigerian teenagers and the ruling family of Saudi Arabia use Facebook or Twitter at the exact same service level, with the same user interface. If you want to write a document, you can use Google Docs, which is what Google’s internal teams use during a crisis. Whoever you are, you can use the same tools, for free, as the people who build the tools.

You can raise money on the same platforms as celebrity musicians or basketball players (Kickstarter, Indiegogo). The Encyclopedia Britannica used to cost upwards of $1,500. Today, Wikipedia provides far more knowledge, for free, for everyone, everywhere. You can rent as many powerful computers as you want, for $1/hour (Linode, Digital Ocean). As the joke goes, on the Internet, no-ones knows you’re a dog.

Technology, and particularly software, have been an incredible equalizing force.

If you have $200 for a computer (or much less for a used model), and occasional access to both the Internet and electricity, you can use, for free, the same tools that a 20-year computer industry veteran such as myself uses (Ubuntu Linux, Go). You have access to all the same learning materials as I do. You can participate in the same forums, communicate with the same people, on the same terms. In the words of the Hacker Manifesto: “If it makes a mistake, it’s because I screwed it up. Not because it doesn’t like me…”.

When you stop to think about it, this is unprecedented. Powerful, affluent, in-group people have always had better homes, transport, and food, breathed better air, and lived longer lives, than the less powerful. There is almost no other facet of human existence where we are all treated exactly equally. It is debatable whether humans, in the physical world, are even capable of this.

All this is possible, maybe inevitable, because of the tools that we in the software industry have built, because of the Internet, and the Open Source model. It makes me incredibly proud to call myself your equal.

July 24, 2015

Building shared libraries in Go: Part 2

Posted in Software at 22:12 by graham

In part 1 we called a very simple Go shared library from Python. Let’s do a more complex example, passing string and []byte, from C++, and getting back a []byte.

Calling Go from C++

Save the following as concat/main.go:

package main

import "C"

//export Concat
func Concat(sIn string, bIn []byte, bOut []byte) {
    n := copy(bOut, sIn)
    copy(bOut[n:], bIn)
}

func main() {}

We add the import "C" so that cgo gives us a header file to #include. Build the shared library and header:

go build -buildmode=c-shared -o libconcat.so concat

In libconcat.h we have this signature:

extern void Concat(GoString p0, GoSlice p1, GoSlice p2)

GoString and GoSlice are defined further up in the header file like this:

typedef struct { char *p; GoInt n; } GoString;
typedef struct { void *data; GoInt len; GoInt cap; } GoSlice;

Copy libconcat.so to /usr/lib/ (or wherever your libraries live). Now let’s call Concat from C++:

#include <vector>
#include <string>
#include <iostream>
#include "libconcat.h"

int main() {
    std::string s_in {"Hello "};
    std::vector<char> v_in {'W', 'o', 'r', 'l', 'd'};
    std::vector<char> v_out(11);

    GoString go_s_in{&s_in[0], static_cast<GoInt>(s_in.size())};
    GoSlice go_v_in{
        v_in.data(),
        static_cast<GoInt>(v_in.size()),
        static_cast<GoInt>(v_in.size()),
    };
    GoSlice go_v_out{
        v_out.data(),
        static_cast<GoInt>(v_out.size()),
        static_cast<GoInt>(v_out.size()),
    };

    Concat(go_s_in, go_v_in, go_v_out);

    for(auto& c : v_out) {
        std::cout << c;
    }
    std::cout << '\n';
}

Save that as concat.cpp. Copy libconcat.h into the same directory. Build and run:

g++ --std=c++14 concat.cpp -o concat -lconcat
./concat

I need the static_cast<GoInt> because a GoInt is a signed type (long long on my machine), but size() returns unsigned type size_t. Apart from wrapping things in Go[String|Slice|etc], this is exactly like calling an ordinary shared libary, because that’s what we built, an ordinary shared library.

The one very important caveat is that you should not pass pointers to Go allocated memory back to the C++ side; that’s why we use an output parameter (v_out). Even if you maintain a reference to it on the Go side so that the garbage collector doesn’t reclaim it, the Go runtime reserves the right to move that memory if they build a copying garbage collector. Lots of details in issue #8310.

Passing C++ allocated memory to Go is fine. Go will not garbage collect memory it did not allocate. There rules here are the same as for cgo.

July 15, 2015

Building shared libraries in Go: Part 1

Posted in Software at 03:22 by graham

Since 1.5, we can write C-style shared libraries (.so) in Go. These can be used directly from C and C++, but also from any language that can use a C shared library, which is most languages. Here’s the original design document. Let’s do a very simple example and call it from Python. In Part 2 we’ll do a more complex example and call it from C++.

Calling Go from Python

package main

import "C"

//export DoubleIt
func DoubleIt(x int) int {
        return x * 2
}

func main() {}

Save that in your GOPATH as doubler/main.go. The exported symbols (the functions the shared library provides) must be in a main package, with a required but ignored main function. They must be marked with magic comment //export <name> (no space), because go build will call cgo.

go build -o libdoubler.so -buildmode=c-shared doubler

This puts libdoubler.so in your current directory. Let’s check we really have a dynamic shared library:

$ file libdoubler.so 
libdoubler.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked ...

$ nm -D libdoubler.so | grep "T DoubleIt"
000000000005d230 T DoubleIt

Well that looks good. Let’s call it from Python, because Python’s ctypes makes this incredibly easy:

>>> import ctypes
>>> lib = ctypes.CDLL("libdoubler.so")
>>> lib.DoubleIt(21)
42

Ta da!

For me this ability to build normal shared libraries makes Go a first-class language on Unix. In later posts we’ll do a more complex example with C++, and time permitting a SWIG example with Ruby.

July 10, 2015

How memory is allocated

Posted in Software at 16:38 by graham

tl;dr man 2 brk

Last year when I was learning assembler, I was asking myself how to allocate memory without malloc. Usually memory is either allocated for us by our language, or we do it with new or malloc. But malloc is a library function, it’s not a system call. How does malloc itself get memory from the kernel? To answer that we need to look at the layout of a program in memory.

On Linux amd64, every process gets it’s own 128 Tb virtual address space. The program code, global data, debugging information and so on are loaded at the bottom of that space, working ‘upwards’ (bigger numeric addresses). Then comes the heap, where we are going to allocate some memory. Where the heap ends is called the program break. Then there is a very large gap, which the heap will grow into. At the top of the address space (0x7fffffffffff) is the stack, which will grow downwards, back towards the top of the heap. Here is a graphic of virtual memory layout

To allocate memory on the heap, we simply ask the kernel to move the program break up. The space between old program break and new program break is our memory. The system call is brk. First we have to find out where it is now. brk returns the current position, so we simply have to call it. We pass it 0, which is an invalid value, so that it doesn’t change anything.

    mov $12, %rax   # brk syscall number
    mov $0, %rdi    # 0 is invalid, want to get current position
    syscall

When that returns, the current position is in rax. Let’s allocate 4 bytes, by asking the kernel to move our break up by four bytes:

    mov %rax, %rsi  # save current break

    mov %rax, %rdi  # move top of heap to here ...
    add $4, %rdi    # .. plus 4 bytes we allocate
    mov $12, %rax   # brk, again
    syscall

We can now store anything we want at the address pointed at by rsi, where we saved the start of our allocated space. Here is a full assembly program which puts “HI\n” into that space, and prints it out. alloc.s. Compile, link, run:

as -o alloc.o alloc.s
ld -o alloc alloc.o
./alloc

To free memory, you do the opposite, you move the break back down. That allows the kernel to re-use that space. Happy allocating!

June 30, 2015

Go: Slice search vs map lookup

Posted in Software at 07:09 by graham

tl;dr Use a map.

Computer science 101 tells us that maps are constant time access, O(1). But what is the constant? The map has to compute the hash, find the right bucket (array access), the right item within the bucket (another array access), and potentially do that multiple times as we walk a chain of buckets (if we overflowed the original bucket). At what point is it faster to iterate through an array comparing each item until we find the one we want?

Motivated by this comparison in C++ I decided to compare Go’s built-in map and slice types. The code is in a gist, a simple set of benchmark tests.

I tried two cases:

  • first a traditional key-value setup comparing map[string]string with []*Item{string,string}. The break-even point here is just five items. Under that the slice is faster, above it is slower.
  • second a set of integers, comparing map[int]struct{} with []int. The break-even point is ten items.

These results are similar to the C++ results. They mean we won’t be doing clever hacks in the name of performance. I call that good news. Use the obvious data structure and it will also be the right choice for performance.

Read the rest of this entry »

June 19, 2015

Software engineering practices

Posted in Software at 03:08 by graham

A selection of software engineering practices, from notes I took at an XTC meetup many years ago. They have survived the test of time very well. Practices change much slower than tools, and are typically a better investment.

  • Keep your methods short.
  • System should always be running – this means you need either live upgrades or a cluster.
  • Always have a goal in mind – visible task. For example keep you current task on an index card stuck to the side of your monitor.
  • Code should express intent – self documenting code.
  • Verbose test names – Think of test methods names as a sentence.
  • The code is the design.
  • Do not duplicate code, effort, anything.
  • Twice is a smell, three times is a pattern.
  • No broken windows. Fight entropy.
  • Check everything into source control. Tools, libraries, documentation, everything.
  • Separate things that change.
  • Write pseudo code first and keep it as comments if it is still useful. OR, write pseudo code as methods and then flesh out.
  • Code for the common case – focus on building functionality that is the most useful first. 80% value for 20% effort.
  • Text is king.
  • Have courage – don’t talk instead of doing – resolve design debates by implementing the alternatives.
  • Take a break.
  • Proceed in small steps / small iterations.
  • Understand the domain. Better domain understanding lowers communication cost and means you can use vaguer specifications.
  • Make it work, then make it right, then make it fast.
  • Strive for a coherent abstraction – does each ‘unit’ fit together.
  • Open black boxes – look into bits of legacy systems you don’t know.
  • Automate.
  • Avoid ‘Manager’ objects – instead make them more specific: Retriever, Calculator, Finder, etc.
  • Think in terms of interfaces, of behaviour, not of data.
  • No magic: No wizards.

June 18, 2015

How I read job postings

Posted in Behaviour at 00:40 by graham

In the interests of illustrating the complicated programmer psyche for the benefit of anyone involved in recruitment, here’s the two things I look at in a job advert:

  • Is it using a programming language / technology that I know and enjoy?

    If I don’t enjoy the tools, the days are really going to drag. If I don’t already know the language, you’re probably not going to hire me.

  • Is it fully remote, or near my house?

    I have to live somewhere, and right now I live here. I’m sure your office is lovely.

Those are the gatekeeper questions. If the answer to either of those is no, there’s no point reading on, it’s not going to work. You won’t want me, or I won’t enjoy the work, or I’ll have a nasty commute / relocation. If the answer to both of those is yes, then I can get excited, it’s a real opportunity.

Job ads too often focus on what matters to the person writing them, not what matters to the person reading.

May 27, 2015

Go: The price of interface{}

Posted in Software at 00:43 by graham

Go’s empty interface{} is the interface that everything implements. It allows functions that can be passed any type. The function func f(any interface{}) can be called with a string f("a string"), an integer f(42), a custom type, or anything else.

This flexibility comes at a cost. When you assign a value to a type interface{}, Go will call runtime.convT2E to create the interface structure (read more about Go interface internals). That requires a memory allocation. More memory allocations means more garbage on the heap, which means longer garbage collection pauses.

Read the rest of this entry »

May 13, 2015

Science fiction: Ancillary Sword

Posted in Society at 04:56 by graham

It was a gesture meant to comfort. Even if they hadn’t already known the reason for our coming, they could not have failed to notice my shaved head and the mourning stripe across my face, and Captain Hetny’s. These people didn’t know us, quite possibly didn’t know who had died. We represented the forces that had conquered them, torn them away from their home world to labor here. They had no reason to care for our feelings. They had no reason to think that either of us knew enough Delsig to understand the words. And no expectation that we would understand the import of their song even if we did. Such things are fraught with symbolic and historic significance, carry great emotional weight – but only for someone aware of that significance to begin with.

They sang it anyway.

From Ancillary Sword, by Ann Lecke.

If you’re going to read it, start with Hugo and Nebula away winner Ancillary Justice.

March 2, 2015

Quotes from veteran software engineers

Posted in Software at 07:53 by graham

The software industry is the most fashion-conscious industry I know of.
– Ivar Jacobson
If you take some of the programs that exist today, they are more complex that just about any artefact that humankind has build before.
– Bertand Meyer
The best path to high-quality software is talented experts who share a pretty clear sense of what they want to produce. I have no idea how to produce good software without talented programmers.
– Peter Weinberger

Masterminds of Programming is a fascinating book, where the authors get many of our industry’s most highly regarded veterans to speak out about basically anything software related. The hook is that they’re being interviewed about the programming language they created.

Here are some of the best quotes and most interesting historical anecdotes.

Read the rest of this entry »

« Previous Page« Previous entries « Previous Page · Next Page » Next entries »Next Page »