Richard Jones, Esq.

Transcoding HTTP mp3 streaming proxy in bash

Here’s how to make a proxy for streaming mp3s. It transcodes on-the-fly to 64kpbs MP3 using lame. When transcoding is finished, it calls the ./posthandler.sh script, which can either just delete the file, or potentially archive it so you don’t need to transcode it again.

One interesting limitation seems to be the buffer size of a fifo pipe in linux. Even though the transcoding step is pretty quick, if a client is connected the transcoding only manages to fill the pipe a couple of hundred k ahead of what is being read.

The -i flag to `tee` means it ignores interrupts, and will finish transcoding the file and call the posthandler even if the client disconnects.

Run is like this:

while [ 1 ]; do nc -vlp 8080 -c './transstreamer.sh' ; done

Then hit up a url of your choice using your awesome new proxy:

mpg321 "http://localhost:8080/http://freedownloads.last.fm/download/105468518/Letters%2BFrom%2BThe%2BBoatman.mp3"

Not the most scalable solution, but a mildly amusing quick hack.

Tags: bash, hack, netcat, streaming

Erlang libketama driver – Consistent Hashing

All the data I need from memcached is assigned to servers using a consistent hashing mechanism, implemented as libketama – a shared library written in C. We use a php extension to wrap this, and also have a pure java implementation. Rather than port the algorithm to Erlang, I wrote a an Erlang driver.

There are 3 things covered here:

• A small driver program written in C (using libketama)
• Some basic testing from the shell using Perl and xxd
• The Erlang gen_server that calls it

C driver program

Testing the driver with Perl and xxd

Before writing the Erlang bit, it’d be nice to know the driver program does what we expect.  Will send the driver a 1-byte length header followed by the key, and expect a 1-byte length header and the value as a response. Say we’re hashing a memcached key ‘user:123′ to a server, we can do what the Erlang port does with a bit of perl, and the ‘xxd’ command to see output in binary.

perl -e '$key="user:123"; $len=pack("C",length($key)); print $len; print $key;' | xxd -b

0000000: 00001000 01110101 01110011 01100101 01110010 00111010  .user:
0000006: 00110001 00110010 00110011                             123

Note the first byte (00001000) printed before the key is the length of the key, 8. Now let’s send this to the driver program and check the response (provide a valid ketama.servers file):

perl -e '$key="user:123"; $len=pack("C",length($key)); print $len; print $key;' | ./ketama_erlang_driver /var/ketama.servers | xxd -b

0000000: 00010000 00110001 00110000 00101110 00110000 00101110  .10.0.
0000006: 00110001 00101110 00110001 00110001 00111000 00111010  1.118:
000000c: 00110001 00110001 00110010 00110001 00110001           11211

The first byte of the response (00010000) is 16, which is the length of the server address returned by the driver, “10.0.1.118:11211″ – It does what we expect, onwards…

The Erlang bit

This code can be found in the erlang directory of the ketama source in svn:
svn://svn.audioscrobbler.net/misc/ketama/

Tags: c, driver, erlang, hashing, ketama, memcached

Reading Serialized PHP Objects from Erlang

I started writing some Erlang recently. The vast majority of data I need to access from Erlang resides in cached, serialized php objects. Here’s what I came up with to turn a serialized php object into a sort of nested Erlang proplist thing.

This gives:

a:4:{i:0;i:123;i:1;s:5:"hello";i:2;d:3.14;i:3;a:2:{s:1:"a";s:3:"foo";s:1:"b";s:3:"bar";}}

It’s not hard to see how the (relatively undocumented) PHP serialization format works. Here’s what it becomes in Erlang:


1> php:unserialize("a:4:{i:0;i:123;i:1;s:5:\"hello\";i:2;d:3.14;i:3;a:2:{s:1:\"a\";s:3:\"foo\";s:1:\"b\";s:3:\"bar\";}}").

{[[{0,123},
{1,<<"hello">>},
{2,3.14},
{3,[{a,<<"foo">>},{b,<<"bar">>}]}]],
[]}


Here’s what it does with objects:


2> php:unserialize("O:12:\"ExampleClass\":3:{s:2:\"id\";i:123;s:4:\"name\";s:2:\"RJ\";s:9:\"languages\";a:3:{i:0;s:3:\"php\";i:1;s:6:\"erlang\";i:2;s:3:\"etc\";}}").
{[{class,"ExampleClass",
[{id,123},
{name,<<"RJ">>},
{languages,[{0,<<"php">>},
{1,<<"erlang">>},
{2,<<"etc">>}]}]}],
[]}

Due to a combination of PHP’s “relaxed” type system, an old database abstraction library, and munging things in and out of memcached, we sometimes end up with numeric properties, such as ‘id’, represented as strings by PHP. To mitigate this, I ended up with some nasty code that forces certain properties to a predefined type (“id” is always an int, etc..). Yuk. Anyway, here’s the Erlang module:

Tags: erlang, php