I’ve recently finished reading Getting Things Done. And I’m trying it out as my method for organising all of my work. Regardless of how it goes there are several key ideas which I think transfer directly onto the world of software development. These are (in no particular order):

• If something takes less than 2 minutes do it,
• Stuff that needs doing is defined as something that isn’t how or where is should be.
• A project (something that needs planning) is something that requires more than one action to complete
• Always think in terms of “What’s the next action?”

This last one I consider crucial especially when it comes to learning new things. I’ve been guilty of having goals such as “learn functional programming”. This type of thing is far too broad, you need to think it through until you end up on the next thing that needs doing to reach your goal. All of us are guilty of having a number of projects “on the go” as well. I’ve found forcing myself to think about the next action has really helped me make progress with them. In some cases I’ve made progress on projects that haven’t made progress in >2 years. Like everything, GTD is no silver bullet, and isn’t for everyone, but I think the book is worth reading, if you don’t find it useful, at least you know.

I don’t normally find programming koans enlightening or amusing, there is one execption – which pretty much sums up my approach to vim:

1
2
3
4
5
6
7
8
9
10
11
12
13

Master Wq and the Markdown acolyte

A Markdown acolyte came to Master Wq to demonstrate his Vim plugin.

"See, master," he said, "I have nearly finished the Vim macros that translate Markdown into HTML.
My functions interweave, my parser is a paragon of efficiency, and the results nearly flawless.
I daresay I have mastered Vimscript, and my work will validate Vim as a modern editor for the enlightened developer!
Have I done rightly?"

Master Wq read the acolyte's code for several minutes without saying anything. Then he opened a Markdown document, and typed:

:%!markdown
HTML filled the buffer instantly. The acolyte began to cry.

But I was starting to get the feeling that most vim users had loads of plugins installed – so I decided I wanted to find out if this was true.

20 minutes later I had hacked my way through a basic survey – which was full of holes and typos. Despite this I had over 1000 responses in 24 hours – which gives us our first ‘statistic’:

1

Vim users love talking about vim

So, here are the results. 1300 of you responded to the survey. Percentages have been rounded down to the nearest integer. Because I said so. Questions where the answers were freeform I’ve done my best to group. But it might not be perfect. Each question was optional so each question had a number of empty responses, I’ve left these out. This is why most percentages do not add up to 100%

I welcome others to take a look at the raw results: Link to Raw Data

Do you spend most of your time in vim or gvim?

I forgot about macvim, and had never heard of vim-qt – but thankfully most of you seemed to work out that my intention was – gui or terminal.

vim 946 72%

gvim 347 26%

How often do you use vim?

daily 1227 94%

weekly 54 4%

monthly 11 0.8%

The obligatory operating system question – Which do you use most?

The world is full of programmers using macs in coffee shops right?

linux 807 62%

os x 347 26%

windows 131 10%

freebsd 19 1%

One person said they used “your face” which is an odd choice as my face has very poor memory management.

What key do you have mapped as your leader key?

I’ve always just stuck with the default, althought I was aware that alot of people use ‘,’. Using space was new to me, but according to this it’s one of the most popular keys

comma 470 36%

backslash 309 23.7

none/ don’t use, never heard of leader 220 17%

space 169 13%

semicolon 13 1%

forward slash 12 0.9%

esc 11 0.8%

colon 8

underbar 5

ctrl 4

alt/ caps lock 2

How often do you make changes to your vimrc file?

montly 603 46%

weekly 459 35%

yearly 138 10%

daily 60 4%

no customisations 21 1%

When did you first use vim?

2010’s 613 47%

2000’s 505 38%

1990’s 162 12%

How many plugins do you have installed at the moment?

This is the question I really wanted answered. Thankfully it seems like most of us would rather have less plugins

Your favourite plugin?

ctrlp 171 13%

syntastic 59 4%

nerdtree 59 4%

youcompleteme 50 3%

fugitive 41 3%

airline 39 3%

surround 30 2%

vundle 30 2%

pathogen 24 1%

Do you remove plugins you no longer user?

yes 1009 77%

no 229 17%

What is your average typing speed?

61 – 80 391 30%

41 – 60 360 27%

> 80 344 26%

20 – 40 wpm 121 9%

< 20 wpm 8 1%

Roughly how many different computers will you use in a particular week?

6 – 10 143 11%

2 – 5 877 67%

> 10 122 9%

1 132 10%

Have you written a plugin?

no 1006 77%

yes – publicly available 121 9%

yes – kept it private 148 11%

Give me a single piece of advice

Some of it is great, some is crazy, you’re welcome to take a look at the raw results:

It’s all here

Your view of emacs?

never used emacs – don’t want to 498 38%

never used emacs – but willing to try 418 32%

i use it every now and again 221 17%

it’s another tool in my toolbox 113 8%

I spent a while trying to work out how to construct this question, what I was trying to work out is if the editor war really existed, I don’t use emacs, but I did, and there are some things I think it does better than vim, and some things it doesn’t. I spent ~2 years using emacs before moving to vim, I wanted to get a broad idea of how emacs is view by the vim community. The results were a suprise

That’s it

That’s all there is nothing more. I’m not great with statistics so I encourage others to take a look at the raw data and come up with a much better ‘analysis’. If anyone feels that the above is in some way worthy of a small tip here’s my btc address:

1CQJ4VkFgV45NnwETRZnHsbZq6djN9YRGH

Update: So I’ve learned that there’s actually some work to be done here around sandboxing before it’s actually ready for public use. Until then I’ve removed the server, but the code is still available if you want to play around with it in a trusted setting.

One of the great things about go is the whole environment you get with it, gofmt, godoc, go get. It’s all good stuff. There’s also a playground for you to try things out on. The docs also have runnable examples in them based off a similar idea: http://golang.org/pkg/encoding/json/#example_Unmarshal

It’s a great idea, it means you can actually try the code out while you’re reading the docs. But it only works on code from the stdlib. But that makes sense doesn’t it? You don’t want to allow anyone to import some arbitrary package.

But it occured to me that when someone develops a package, the author might want to include runnable examples for their own code.

The source for something similar to the playgrond is incuded in the go release (misc/goplay) I’ve modified it very slightly to allow /compile to be called from any location. I’ve called it goplaas (It’s not really as a service, but that’s the standard suffix these days isn’ it? And it got your attention!)

The idea is you install it on your own server, and it will have access to anything installed on your GOPATH. This means you can control what packages can be imported, and also the version.

All you need to do is put some javascript and a textarea into a webpage, point it to your server and you’re done.

Here’s an example using the loggo package from launchpad.net. Try it

1

I've removed it - sorry

I’ll try to keep my server running with goplaas for the next few weeks. If it suddenly stops working, it’s likely that I’ve pulled the server down. You can experiment yourself by running

1
2

go install github.com/mattyw/goplaas
./goplaas -http :9999 #Listen on 9999

goplaas includes a -compile flag for sending compilation instructions to another server, full instructions are in github.

A good next step would be to package this all up into a docker container to allow anyone to install it with relative ease

Today I finally got round to trying out something i’d been meaning to look into for nearly a year, and seeing as clojure and editors seems to be a topic again I thought it would help add some diversity.

Acme is an editor unlike any other. It was designed by Rob Pike for Plan9 (although it’s available for unix-like systems using Plan9 from User Space). It’s designed to be an editor and shell. Unlike emacs and vim which encourage you to keep your hands on the keyboard, acme requires use of a 3-button mouse, and it supports something called mouse chording. A rough guide would be:

• Button 1: Select Text
• Button 2: Execute Command
• Button 3: Search/ acquire text (or load file)
• Button 1-2 Cutting
• Button 1-3 Pasting
• Button 1-2,3 Cut+Paste (Copy)

My introduction to acme was via Russ Cox’ video, which is a great starting point:

You can probably summarise the whole thing by saying that in acme – text is data. Once you type some text, it can be saved into a file, executed as a command or passed as an argument to a command.

Obviously this resonated with the clojure part of my brain. It occured to me that being able to select some clojure code in acme and evaluate it would make acme a great editor for clojure.

Thanks to nrepl the job is really quite easy. I wrote a very very simple nrepl client in go. You can find it on my github

All it does is read clojure code from stdin and send it to nrepl. This is really all there is to do. In acme you just need to type your command in one of the scratch areas:

Take a look at the below screenshot. You can see:

1. In yellow: selected clojure code with mouse button 1
2. In red: selected command with mouse button two
3. Release the buttons and the window in the bottom right contains the evalutated code

Here’s a video of me doing it:

Clojure in Acme from Matthew Williams on Vimeo.

To get this setup yourself:

1. Install plan9 from user space to get acme
2. run lein repl in some terminal
3. You’ll need a command line tool that can take clojure code on stdin and send it to nrepl. I used my own gonrepl tool- To use that you need to install go then run:

1

go install github.com/mattyw/gonrepl/...

That’s it. Thanks for reading!

tl;dr: Juju is aimed at deploying and scaling hundreds of services. But it’s also a handy alternative to buying a VPS.

I don’t have a VPS, I have in the past but I’ve never used them enough for me to be able to justify the cost long term. Juju provides a perfect solution to this use case.

I have a very simple charm I call devenv, it doesn’t do much more than install stuff I use for doing development, like installing tmux and grabbing my configuration files. I’ve blogged about it previously here.

This week I was having trouble getting a client library to connect to a simple server program I’d written, I was lost with debugging but luckily the library author volunteered to help me out. I already had a binary of the server that could be used to generate the error in the client library, but I don’t have a VPS, so what’s the quickest way for me to make the server public so the library author can play around with it?

Juju!

All I needed to do was to add the following line to the start hook of my devenv charm:

1

./app &

And copy the app binary to my charm directory.

It’s sort of an anti pattern – but because juju copies the whole charm dir when it deploys, it will grab any binaries in the charm dir as well, it’s not a great idea to make use of this for serious deployments, but for me it was great

With that done all I had to do was deploy it:

1
2

juju bootstrap
juju deploy --repository=charms local:precise/devenv

From deciding I wanted to deploy the binary – to having it deployed was under 5 minutes. 20 minutes later the library was fixed and the server was taken down

1

juju destroy-environment

That was it. All in all it cost around $0.02 (that’s just over 1p). Much better than having a VPS that would largely sit idle.

Of course the other advantage with using juju is that I’m not tied to a particular cloud provider, so I can choose which one I want depending on the pricing at that moment. And since juju now supports local mode you can do your charm development offline to save even more money!

Great, how does this help me?

If you’re thinking of getting a VPS – but don’t feel you can justify the cost because of the amount of use you’d get out of it, then you really should try out juju. Once you’ve got it installed there’s a handy charm that just deploys ubuntu server, it’s a great starting point for writing a charm to setup your own development environment.

1

juju deploy ubuntu

Here’s another link to my post about writing a charm to build a development machine: http://mattyjwilliams.blogspot.co.uk/2013/05/deploying-development-machine-with-juju.html

Don’t write macros is a pretty good rule, but when you’re learning a language it’s quite a fun little exercise. One such exercise I like to try is this:

Write a function so that I can call system commands without having to wrap them in quotes.

So rather than doing this:

cmd([“grep”, “-rin”, “foo”, “./”])

I can do this:

cmd([grep, -rin, foo, ./])

Clojure

In lisps like clojure, the solution is very elegant, thanks to the minimal syntax, you end up with something like:

1

(call grep -rin foo "./")

And the macro to do it is also pretty elegant:

1
2
3

(defmacro mcall [& args]
  (let [sym (gensym)]
  `(apply sh (map str '~args))))

Not too much magic (as far as macros are concerned) here. Make a unique symbol with (gensysm) to keep our macro hygienic. We syntax quote our whole list using ` to not evaluate anything unless we say so. That means we have to do some magic with our args parameter: ‘~args. ~ here means evaluate args (so we get the symbols we passed in) but then we quote with ’ so that we don’t evaluate these symbols. That’s the magic that lets us not need quotes around our arguments. The rest of the code just puts them in the right place by mapping our args list against the str function and applying it to sh. This is how you use it:

1
2

(prn (mcall ls -l))
(prn (mcall grep -rin not "./")) ; We still need quotes around ./ because it's not valid syntax

By coincidence I blogged about clojure macros almost exactly one year ago: http://mattyjwilliams.blogspot.co.uk/2012/08/another-clojure-macro-tutorial-that-no.html

Elixir

Elixir is a lovely little language I’ve been playing with recently, so naturally I wanted to try out the macros. I needed to ask a few questions on the irc channel (thanks ericmj and true_droid!), but I got it cracked:

1
2
3
4
5
6

defmacro mcall(args) do
    call = Enum.map_join(args, " ", Macro.to_string(&1))
    quote do
      System.cmd(unquote(call))
    end
end

Again, not much magic, map_join maps our list using the Macro.to_string function then joins at the end, this is done outside of the quote since we will work it out at compile time. We then pass this to System.cmd in a quote block – which means don’t evaluate this yet, that will happen when we call mcall at runtime.

1
2

IO.inspect MacroTest.mcall([ls, -l])
IO.inspect MacroTest.mcall([grep, -rin, def, "./"]) # We still need quotes around ./ because it's not valid syntax

The idea

We’re going to use a raspberry pi and juju to setup a wordpress blog on aws.

The raspberry pi is an amazing little gadget. Low power enough that I don’t feel guilty about leaving it turned on overnight, but flexible enough that I can shove leds into it to grab my attention if needed.

Juju is ubuntu’s answer to setting up services in the cloud. Once installed you simply setup your credentials with an existing cloud provider like aws or hp cloud and then deploy things into it. Setting up wordpress for example is as simple as

1
2
3
4
5

juju bootstrap # Setup the cloud environment
juju deploy wordpress # Start an instance and install wordpress on it
juju deploy mysql # Start an instance and install mysql on it
juju add-relation wordpress mysql # Connect wordpress and mysql together
juju expose wordpress #Allow wordpress to be accessed from a public ip

The Plan

1. Install go onto the raspberry pi
2. Install juju
3. Deploy a wordpress blog

We’re going to be installing the latest version of juju which is written in go.

Installing Go

Dave Cheney has written an excellent article on Installing go on the raspberry pi. On my model B (1.0 revision) I had to update raspi-config so that I could select the 240M/16M memory split, but you might have to see how you get on here. I choose to skip the tests and just ran ./make.bash to build go rather than all.bash.

I was unable to build juju using go 1.0.3. I ended up building go 1.1.1 from source, which seemed to work fine. From memory I believe you need to setup a folder to be your GOPATH before you can install the latest version of go.

Installing Juju

Getting the latest tip of juju-core is easy using the go get command, it grabs all the dependencies for you, it does mean we’re grabbing tip, but there are ways to fix that later if we need to. juju-core is hosted on launchpad.net which uses bazaar for source control. This means you need to install it before you can run go get. I was able to install it on raspian using

1
2

sudo apt-get update
sudo apt-get bzr

Without doing the update I was unable to grab all of bzr’s dependencies, so I recommend running the update first.

Once bzr is installed you can install juju using go get:

1
2

sudo apt-get install libcurl4-gnutls-dev
go get launchpad.net/juju-core/...

When it’s finished you should find a juju binary installed. Mine ended up in my GOPATH, I think because I had something setup wrong somewhere. But if you’ve got this far I’m sure you’ll be able to find it.

Setting up Juju

There’s much better instructions for setting up juju here The basic steps come down to

1. Setting up a public key
2. Generating an environments.yaml file
3. Filling the file in with the data for your cloud provider

I configured juju for aws. My ~/.juju/environments.yaml file looked like this:

1
2
3
4
5
6
7
8

default: amazon
environments:
  amazon:
    type: ec2
      access-key: MY AWS ACCESS KEY
      secret-key: MY AWS SECRET KEY
      control-bucket: mattyw-ec2-bucket
      admin-secret: mattyw-ec2-admin

Once you’ve done this, try running juju bootstrap. If it finishes without error wait a minute or so and run juju status, If all goes well you should see something like this

1
2
3
4
5
6
7
8
9

$ juju status
machines:
  "0":
    agent-state: started
    agent-version: 1.11.0
    dns-name: ec2-23-20-228-111.compute-1.amazonaws.com
    instance-id: i-0bfc3d64
    series: precise
services: {}

Now we can start the business of getting a blog up and running:

1
2
3
4

juju deploy wordpress
juju deploy mysql
juju add-relation wordpress mysql
juju expose wordpress

You might have to wait a few minutes for everything to start up, keep taking a look at the output of juju status until you see all the agent-states running and expose true on wordpress.

When it’s done you should see something like this in your juju status output:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

 wordpress:
    charm: cs:precise/wordpress-15
    exposed: true
    relations:
      db:
      - mysql
      loadbalancer:
      - wordpress
    units:
      wordpress/0:
        agent-state: started
        agent-version: 1.11.0
        machine: "1"
        public-address: ec2-54-234-76-38.compute-1.amazonaws.com

Now point your web browser to the url in public-address and there you go, you’re deploying stuff from your pi.

Juju doesn’t stop at just deploying blogs. Check out Juju to see what else it can do for you

My journey to elixir

Erlang was the first functional language I ever used. When I first started reading Programming Erlang by Joe Armstrong I was hooked, I was fascinated in a language where “=” didn’t indicate an assignment, and the idea of pattern matching immediately struck a chord with me, I loved the one module per ‘process’ thing. Erlang was set to become my favourite language, there was only one problem.

I didn’t like the syntax

It’s a completely personal view, I know people who love the syntax, I just didn’t. To me it just didn’t have what Christopher Alexander in “The Timeless way of building” calls “quality without a name”. I suppose I just didn’t find it fun to type. So after re writing a couple of internal libraries in erlang to see what they looked like (they were much shorter and quite easy to read) I stopped using it, and started playing around with other languages like haskell and clojure.

Clojure was my first lisp dialect and I enjoy it alot, the syntax, the whole homoiconicity thing, macro’s aren’t something you should use all the time – but it’s comforting to know they’re there to fall back on if you need them.

Parallel to this functional language experimentation was another path – languages I used at work, which has seen me recently move from writing mostly in Python to Go. Go’s channels and go routines are great, but when I started writing go code seriously (about 8 months ago) there was something that immediately stuck me.

1
2
3
4

thing, err := someFunction()
if err != nil {
    //handle the error
}

Now, I’m not going to start complaining about go’s error handling, because I like it. But sometimes I would like to be able to make use of pattern matching and do something like this:

1

{thing, nil} = someFunction()

And have my goroutine just dies if it fails. I know that’s not go’s philosophy with errors, and that’s fine. I’m just saying.

Work and Fun Collide

I’ve spent the past 6 months saying to people that I would like a lisp dialect to be built onto go. What I really meant was I wanted a language that would let me do go style concurrency and play around with a repl, macros and all that functional programming fun stuff.

It looks to me that elixir provides exactly that.

Just look at these two programs. Both call a 10 second sleeping function on a seperate ‘thread’ and wait for the response. The first in go, the second in elixir.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

package main

import (
  "fmt"
  "time"
)

func longRun(response chan string) {
  time.Sleep(10e9)
  response <- "done"
}

func main() {
  response := make(chan string)
  go longRun(response)
  value := <-response
  fmt.Println(value)
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

defmodule Main do

  def longRun(pid) do
    :timer.sleep(10000)
    pid <- {:result, :done}
  end

  def main() do
    spawn(__MODULE__, :longRun, [self])
    receive do
      {:result, value} ->
        IO.puts value
    end
  end

end

Main.main()

They’re not too different, elixir has spawn where go has ‘go’ and elxir has receive where go has select. There’s no denying that they’re completely different languages, in some rather crude tests I’ve done go does seem to be largely faster, but elixir let’s me do meta-programming whilst forgetting about types.

I expect to be blogging alot more about elixir in the future

My first post on my new blog, so let’s keep it simple.

In sql databases you can specify unique constrains on certain columns, like this example from postgres:

1
2

create unique index idx_UniqRef on FooBar (
       UniqRef ASC);

The question is: Is it possible to do this in a schema-less database like mongo? Lets’ start writing some code that we want to fail. For this I’m going to use go and Gustavo’s mgo library

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

+package main
+
+import (
+  "labix.org/v2/mgo"
+  "log"
+)
+
+type FooBar struct {
+  Name   string
+  Age    int
+  UniqId int
+}
+
+func main() {
+  session, err := mgo.Dial("127.0.0.1")
+  if err != nil {
+    log.Fatal(err)
+  }
+
+  coll := session.DB("test").C("uniq_test")
+  coll.DropCollection()
+
+  a := FooBar{"Foo", 1, 1}
+  b := FooBar{"Bar", 2, 1}
+
+  err = coll.Insert(a)
+  if err != nil {
+    log.Fatal(err)
+  }
+  err = coll.Insert(b)
+  if err == nil {
+    log.Fatal("We wrote something but wanted to fail!")
+  }
+
+}

Nothing too fancy here:

• Create a connection to our collection
  • If the collection exists drop it (useful for iterating quickly on this test)
• Create two structures with a UniqId field duplicated
• Insert both, if the second insert doesn’t fail log an error

If we run it as is, we see our failure

1
2

2013/06/12 13:14:51 We wrote something but wanted to fail!
exit status 1

Now we’re in a position to start looking into a solution, and it turns out we don’t have to look far. Just like postgres mongo has indexes, we can create one using ensureIndex. Here’s how it looks in the mgo library:

1
2
3
4
5
6
7
8
9

    //Let's try an index
    index := mgo.Index{
        Key: []string{"uniqid"},
        Unique: true,
    }
    err = coll.EnsureIndex(index)
    if err != nil {
        log.Fatal(err)
    }

This creates an index on our uniqid field and forces it to be unique. If we call this code before we try inserting here’s what we get

1

Failed to write! E11000 duplicate key error index: test.uniq_test.$uniqid_1  dup key: { : 1 }

There we go, it also supports compound indexes which lets you specify multiple fields which must be unique, but I’m not going to cover that here. The mongo documention is a great source for more information