Richard Jones, Esq.

Rewriting Playdar: C++ to Erlang, massive savings

I’ve heard many anecdotes and claims about how many lines of code are saved when you write in Erlang instead of [C++/other language]. I’m happy to report that I now have first-hand experience and some data to share.

I initially wrote Playdar in C++ (using Boost and Asio libraries), starting back in February this year. I was fortunate to be working with some experienced developers who helped me come to terms with C++. There were three of us hacking on it regularly up until a few months ago, and despite being relatively new to C++, I’ll say that we ended up with a well designed and robust codebase, all things considered.

On Feeling Smug

I’ll admit I felt rather smug making it all work in C++ with Boost and ASIO. Getting it to build on all three platforms and dynamically load extensions (DLLs etc) at runtime in a cross-platform way was also quite satisfying (I had plenty of help with that side of things). I learned a lot about C++, Boost, ASIO and CMake. But, as the codebase grew, I began to seriously question my decision to use C++.

My initial reasons for choosing C++ were twofold:

• Distribution – shipping the Erlang VM didn’t sound like fun
• Taglib – *the* library to read metadata from audio files (mp3, m4a, ogg etc) is C++

It turns out Playdar is naturally a good fit for Erlang – it does lots in parallel, and lots of stuff it does is asynchronous and event based. Even with all the stuff you get with Boost, multithreaded stuff in C++ is inelegant, to put it kindly.

SLOCed and Loaded

Anyway, a couple of weeks ago I sat down to re-implement Playdar from scratch in Erlang. I thrashed out the guts of it in a couple of days, and by the end of the week I almost had it 1:1 features with the C++ codebase. There’s still a bit of C++ left – code to interface with taglib.

Using the SLOCcount tool (SLOC=source lines of code) I counted the lines of code in various modules from both codebases, here are the results:

	Erlang Version	C++ Version	Savings
Core Daemon	1,100	4,491	75%
Library + Scanner	197 + 167.cpp	1,355	73%
LAN Resolver	105	427	75%
P2P	463	1,762	74%
TOTAL	2,032	8,035	75%

75% less lines of code using Erlang compared to C++ to implement the same thing – not too shabby :)

The second time around writing in Erlang I knew exactly what I was building, so it’s unfair to compare development time of the two codebases, but given how fast I can type I reckon I saved a good few hours of just pounding the keyboard to input the code (and countless hours of debugging: Erlang tends to work first time, really). Well I’m not sure if “saved” is the right word, considering It was working in C++ already, but it’s my time to waste :)

If you count the third party code bundled with both codebases (excluding boost/asio!) then the erlang codebase saves a whopping 92%. I’m more interested in the savings in code I had to write, however.

Memory and CPU Usage

I’ve done some preliminary comparisons between both projects, when it comes to CPU and memory usage both projects are pretty similar. The Erlang codebase uses slightly more memory than C++ at the moment, but I’m convinced I can get that down to at least as low as the C++ project was. I picked up a few optimization tricks from my three-part Million-user comet experiment in Erlang earlier this year. I’ll post more about this if I learn any new tricks.

One thing I’ve realised about the Erlang codebase is that I’ve used processes to encapsulate state (active queries, specifically) where I didn’t really need to. It seemed sensible at the time, but it’s probably just a waste of memory. I’m going to change it to spawn processes to get the work done (ie, a process that runs the query) but not necessarily just to maintain state.

Distribution to the desktop

C++

You just have to make sure that you build everything and ship with any DLLs along with checks in the installer for system libraries needed (runtime dlls). Oh, and make sure you don’t change the plugin binary interface in the main app, or new plugins will crash and burn when you load them. Add a check for that. Oh and be careful about compiling taglib and stuff with mingw and the rest with VC++, or things might mysteriously crash. Also I heard a horror story about allocating memory in plugin code but deallocating it in the main app when the plugin was compiled against a different stdlib than the main app. This is all par for the course, and the experienced C++ developers I asked for help had no trouble making it work. Size of installable pacakge: 2.5MB

Erlang

Compiling, and building/loading plugins in the Erlang codebase is straightforward on all platforms, as is often the way with VMs. I was against shipping the Erlang VM originally because I figured it would be a lot of hassle and increase the download size substantially. Packaging an Erlang app for the desktop involves taking the installed VM directory structure and stripping out all the docs, source and parts of the Erlang stdlib we don’t use, then packaging it along with the compiled Playdar code. CouchDB does something like this too, and RabbitMQ ships the Erlang VM without stripping unneeded libs. We’ll work on packaging some more (for all platforms), but to date Max has crafted a package that contains the necessary bits of the Erlang VM, a sexy Prefpane to start/stop the daemon on OS X, and the compiled Playdar code all weighing in under 10MB.

We’ll put together a Windows installer soon that’ll probably be around the same size. A 10MB download isn’t so bad nowadays, and I expect we can optimize the packaging process some more. Linux users will get a package that depends on the erlang VM in their package manager.
Seems like shipping Erlang apps to the desktop isn’t so hard after all.

tl;dr

Someone rewrote a C++ app in Erlang: 75% less lines of code for same functionality.

You should read this blog post about Playdar, by Paul Lamere, and take a look at the Playdar website.

C++ codebase (deprecated)
Erlang codebase

Playdar is the future, and the future is written in Erlang :)

Tags: c, erlang, rewrite

Erlang talk at London Hackspace

Last night I gave an “Intro to Erlang” talk at a London Hackspace meetup. I did a quick audience survey first: About 75% did “web programming” (ruby,python,php,etc).  Around 30% admitted to regularly using C/C++/Java or desktop/mobile app development.  Less than 10% had much experience with functional programming.

I wanted to impress upon the audience that Erlang is a practical language, built by Ericsson with a specific purpose in mind. You use Erlang to build useful, scalable and reliable distributed systems in the real world. This was worth pointing out because when many people hear “functional programming” they immediately think of eccentric bearded academics proving the validity of their Haskell code and comparing Monads.

I skipped through the basics of sequential programming in Erlang pretty quickly and tried to spend most of the time showing how you handle processes and send messages. I built a basic Erlang server process that kept a count of how many operations it had done, explaining how it passes state to itself on every loop. Hopefully this helped some people grok how you can build servers that keep a global state by using recursion. I also showed off hot code reloading. We added another feature to the server and upgraded it without stopping it.

You can download the code I used (see link at the end) if you want to try out the examples from last night yourself. The last code I showed was an example of doing the same thing using gen_server, so hopefully if you followed along you’ll have a good understanding of what gen_server is and why it exists.

Hot code reloading example

I can’t write a post about Erlang without including some code, so here’s the basic example I used showing how hot code reloading works:

And if you were following along you saw something like this:

1> c(ex09).
{ok,ex09}
2> Pid = ex09:start().
<0.38.0>
3> Pid ! stats.
Ops: 0, Wtfs: 0
stats
4> ex09:client(Pid, double, 10).
20
5> ex09:client(Pid, triple, 10).
wtf

At this point we added support for “triple” to the example and showed how the fully-qualified call to loop (using the modulename:fun() instead of fun() syntax) causes the newest version of the module to be used:

6> c(ex09).
{ok,ex09}
7> ex09:client(Pid, triple, 10).
wtf
8> Pid ! reload.
Reloading
reload
9> ex09:client(Pid, triple, 10).
30
10> Pid ! stats.
Ops: 2, Wtfs: 2
stats

You can see from the stats at the end that the global state was kept – the server process staying running during the code upgrade.

Download

The slides, example code and basic mochiweb comet project we saw last night can be downloaded here. I should warn you that unless you saw my talk and the various explanations and disclaimers that went along with the code, it’s probably not a good place to start or learn from. Have a look at www.learnyousomeerlang.com or get one of the two excellent Erlang books.

London Hackspace

If you live in London you should know about this. Russ and Jonty (who I worked with at Last.fm for years) started London Hackspace: “We run a dedicated space for people to learn and build things in London.” There are workshops at hackspace meetups on topics ranging from Arduino and electronics hacking, to iPhone development, to Erlang and beyond. Their unofficial slogan could be “Beer & Hacking” – it’s a great place to meet people doing interesting things in London, and to learn new things.

http://london.hackspace.org.uk/

Playdar

Playdar is my pet project at the moment. I talked about this last night too. I wrote it in C++ using Boost, mainly as an excuse to do something serious in C++. I’ve since seen the error of my masochistic ways and in the last week I’ve tossed out the 10,000 lines of C++ and rewritten it in Erlang. I’m not quite finished, but once I have feature parity between the two codebases I’ll write an article comparing the two.  As you might expect, the Erlang codebase is far superior in almost every way.

http://www.playdar.org/

Tags: erlang, hackspace, london, playdar, talk