How we use IRC at Last.fm
Everyone that works at Last.fm is typically connected to our IRC server. We have different channels per team, as well as a company-wide channel, and a few channels dedicated to automated monitoring.
Sometimes it makes much more sense to discuss / ask questions on IRC instead of email, and it’s useful to be able to raise people who are not in the office. That said, the main reason I’m writing this post is to mention the dev-support bot we use: irccat.
IRCCat – Development support bot
The irccat bot joins all your channels, and waits for messages on a specified ip:port on your internal network. Anything you send to that port will be sent to IRC by the bot. IRCCat – as in, `cat` to IRC.
Using netcat, you can easily send events to irc from shell scripts:
$ echo “Something just happened” | nc -q0 somemachine 12345
That will send to the default channel only (first in the config file). You can direct messages to specific combinations of channels (#) or users (@) like so:
$ echo “#syschan Starting backup job” | nc -q0 somemachine 12345
$ echo “#musicteam,#legal,@alice New album uploaded: …” | nc -q0 somemachine 12345
Some of the things we automatically send to appropriate IRC channels:
- SVN commits
- JIRA issue tracker updates
- Nagios alerts for monitored hosts and services
- Deployment notices to testing/staging/production
- Results of automated tests if something bad happens
- Links to pics from security camfeed when someone opens the office door out of hours
We also post messages from automated backup jobs etc, which helps correlate such events with any unusual load spikes or glitches in usually-smooth graphs.
In addition to providing a cat-to-irc conduit, irccat will also hand off commands to a script you can provide. We use this to expose lookup tools and some admin functions to our support staff and developers. The handler script we use is PHP, and has access to our core website libs. Typing “?pokereleasenode”, “?lookup user RJ” or “?uncache artist Radiohead” is faster than writing a throw-away script, more accessible to non-developers, less hassle than a web interface and creates a public log so people can see what’s going on.
The bot is written in Java, it’s easy to build and configure, all the deps are included:
52 Comments to How we use IRC at Last.fm
Leave a comment
About Me
Tags
Recent Posts
- Rewriting Playdar: C++ to Erlang, massive savings
- Erlang talk at London Hackspace
- Anti-RDBMS: A list of distributed key-value stores
- How we use IRC at Last.fm
- Getting to know ejabberd and writing modules
- ssh hack: connect directly to machine via a firewall box
- A Million-user Comet Application with Mochiweb, Part 3
- A Million-user Comet Application with Mochiweb, Part 2
- A Million-user Comet Application with Mochiweb, Part 1
- On bulk loading data into Mnesia
This is pretty awesome. At threadless, the tech team uses IRC to communicate as well – however we don’t use it much for monitoring. I think that your use case is pretty sweet and would be awesome to drop in here. Thanks for sharing the source.
Pretty cool.. I have a similar private irc server setup with a group of about 20 friends (so it’s much more social than business). Been going for over seven years now.
We have a group-wide channel, channels specific to each person (#matt, #mary, etc), and channels specific to topics (#programming, #games, ..).
A variety of bots have been written over the years. Messages can be sent by users to a particular channel or user by emailing channel-[channelname]@example.com or user-[username]@example.com — they are relayed to irc by a bot that is continually checking a mailbox. Email attachments (usually pictures) are automatically unpacked by the bot, posted to a web server, and the URL is posted on irc to the target channel or user.
There are also bots that watch for in-channel messages such as “imdb [film title]” or “book [some book title] [author=lastname]” and performs imdb or amazon lookups. Google SMS is wired in this way, and the “goog” channel command has become quite popular. Also “wiki [topic]” performs wikipedia lookups, posting a summary of the article in channel for everybody to read.
All the channel logfiles are saved forever, and after 7 years of use they have become a treasure trove of information and saved conversations. We have a script for grepping the logs, so for example we can run “greplog foobar” which produces output like this:
#programming.log:2005/11/21-16:59:07 void bob::foobar(void) { extern int sally; /* … */ }
then one can run “logmore #programming.log:2005/11/21-16:59:07″ to browse that logfile at that offset.
Everybody runs irc permanently within a GNU screen session, so there is very little channel joining or parting going on. Instead everybody just connects via SSH and then connects to their screen session, and then catches up on the “scrollback” in each channel.
But my favorite feature is the per-person channels. This is where people write about their day, their thoughts, what they dreamed about last night, what they’re working on, etc. Other people chime in, usually for fun, or sometimes to help. It’s like a microblog — extremely easy to post into, just start typing. But we keep it private within this particular group of good friends. Thanks to these channels, a lot more of our lives have been recorded over the past 7 years than would otherwise have happened. (Yes, we are addicted to this service!)
The text interface within GNU screen leaves a lot to be desired. It would be nice to have the same set of features accessible via web browser. I think that’s coming, slowly. I like what the 37signals folks are doing with Campfire — they have been getting into bots lately. I also like what’s happening on Twitter with Hashtags.
We want a group chat service that is secure, searchable, web-based, and extensible. Various components of that might include Campfire, Twitter/Twitterrific, BitlBee, Hashtags, OTR, CTM…
Whoops, something got munched in my description of the greplog output above. Trying a second time:
#programming.log:2005/11/21-16:59:07<john> void bob::foobar(void) { extern int sally; /* … */ }
[...] How we use IRC at Last.fm [...]
Hey – your quotes look to be broken, I’m only looking quickly at the .commentlist in FF – add width:580px; for a quick fix.
I’m really interested that you guys use IRC, where I work we’ve started trying to get into using Yammer because the majority of people aren’t so tech savy and need something which is more suited for mainstream.
Learning /join /connect etc and a huge gui would be scary for them.
I’m interested to see # in twitter working similar to irc rooms and forums but in a different method. It’s almost like the learning curve of apps like Twitter is designed like pacman/space invaders etc. Insert coin, play. Twitter has a similar philosophy of type message, send.
The learning curve is huge with those games as is Twitter/Yammer etc.
Thanks for sharing!
Out of interest, what irc servers do you guys use?
Sounds awfully similar to what we’ve built at Mochi Media…
We have a bot (hector) written in Python which is basically for amusement purposes, it just responds with markov chain gibberish if you mention its name (and randomly about 2% of the time). This was our first bot :)
Our second bot (juanita) speaks HTTP and is written in Erlang. It’s got monitoring hooks into OpenNMS and a custom app monitoring server that we wrote, svn and trac notifications, forum post notifications, ETL status updates, and a whole bunch of random stuff that we’ve cooked up over the years (ALICE AIML engine, Google queries, DNS lookups, deployment related tools, logging, safe Erlang eval, etc.).
More than half of the company uses IRC on a daily basis, mostly tech people but a few others. It’s not really clear to me that the rest would make good use of it if it was a webapp instead of IRC, I think the noise would be mostly distracting to them. We have some pretty thorough instructions for how to set up Chatzilla or Colloquy for our IRC server, which is pretty easy and you don’t need to know /connect or /join.
Very similar approach to what we use where I work. IRC is an invaluable resource for us, especially when people are working from home.
As for non-technical people using IRC, it’s really not that hard. We have a copy of PJIRC hosted, so non-technical users without an IRC client can just go to that page, click a link and be immediately on IRC. We also have CGI::IRC, just for anyone who doesn’t use Java. However, the majority of staff manage IRC clients with no problems whatsoever.
[...] Jan. 9th http://www.metabrew.com10:59 h How they use ICR at last.fm 0 [...]
[...] This link explains how it’s used at Last.FM. Read the comments below, as they’re just as insightful as the artcile. [...]
@KaiChanVong: I do the machine config here for the non tech savvy users (the Linux ones, unsurpisingly, are pretty good at figuring it out) and any decent IRC client (I use mIRC) will store all that stuff so the user never has to do anything. My standard config includes the server and the secondary server, with automatic retry and auto joining of channels with all the keys so when they come to work, they open the client, and a nice window with lots of tabs with the channels in appear.
Set up all the highlights for them so they know when someone is talking at them and you’re laughing.
It’s a pretty useful means of communication, especially if you want a company wide quick-notification (we get everyone on IRC) as you can choose some good highlight words.. some of our favourites include “pub”, “lunch”, and “shop” ;)
The Mac users here set themselves up too and generally settle with Colloquoy, although I hear a lot of complaints about that. But it will do exactly the same for a user who does not want to know what “/connect” “/join” etc. does.
Most clients have the ability to send any command at connect anyway, so you can also make them /identify… easy to pick out the imposters then!
Hey Jonesy,
I didn’t know you were at Southampton – i thought you were at York. My girlfriend is currently at Southampton. Anyway, congrats on all your successes. We should catch up soon.
Rich
@KaiChanVong:
how about using cgi:irc with a predefined channel join for different users (or even an adapted layout!). The GUI is really easy to learn (just type & send), no installation,, no /commands.
We used this for company internal and external commmunications at a former small firm. Still using it to date to keep in touch with friends.
The logs are a great knowledge repository.
Wow, it’s interesting to hear similar uses at other companies – thanks for the feedback.
Regarding the camfeed, we have a networked Axis security camera and it has an auto-ftp-upload feature, and the ability to send formatted text to an arbitrary ip/port – so we send an announcement with link to IRCcat directly from the camera. It was a fun afternoon when we figured that out :)
@kaichanvong I remember seeing IRCcat talked about at FOWA a few years back by Anil I think it was.
Sadly I think we’re stuck with Team Foundation Server and good old email for the timebeing :(
Sounds good and useful, but what about authentication stuff? I understand from what you write that a bot typically sends the data it got from the port to IRC server without authentication. In this scenario, anyone who knows the IP adress and port of the bot will be able to send messages?
Cool, article. Very inspirational.
This is slick, another use for a tired and true technology. I’ve done monitoring things like this with Jabber and some python scripts, more time to setup, but is similar. If I had a big enough team and everyone used (and knew how to use IRC) this would be ideal. Nice article, got me thinking!
@eren: Correct, but the box running IRCCat is inside our network and not publicly accessible.
[...] by how_gauche to programming [link] [83 comments] Go to Source Share and [...]
FYI, the README says you need sun jvm >=1.5 , but you use some java6 things in the code (String.isEmpty() was the first one that was hit when I tried compiling with 1.5).
After installing sun-java6-{jdk,bin} , it compiled.
I used a similar thing at leadpoint.com…IRC is primarily used for releases so the deployment team can coordinate with engineers. A permanent bot exists that can pull up jira tickets and announce errors. I also created a bot in python that…
shows jira tickets
searches confluence
shows current staging info
each hour it displays stats on database performance and revenues
punishes people…non-engineering types would lurk on IRC to complain instead of using jira so I made the bot demand input every few minutes from certain people.
and it replicated itself and delivered insults…
[...] How we use IRC at Last.fm | Richard Jones, Esq. (tags: productivity hack irc) [...]
[...] How we use IRC at Last.fm | Richard Jones, Esq. [...]
[...] and War By Prof. Peter Dale Scott Where Tijuana meets San Diego Storm Worm botnet cracked wide open How we use IRC at Last.fm Fresh Tracks in Iowa The End of the Christmas Season – The Big Picture [...]
[...] as you could be. I was thinking about this when I read a blog post on how IRC can be a nice low-interruption news source. It can be handy for that, but the real value in IRC is other [...]
[...] http://www.metabrew.com/article/how-we-use-irc-at-lastfm/ : cet article explique comment Last.fm utilise IRC pour faciliter le développement. [...]
[...] How we use IRC at Last.fm [...]
Java? Shame ;-)
No, very interesting article, nice insight into how stuff works.. I’m willing to bet Russ was something to do with all this!
Pulling a few topics together and doing some shameless plugging, you may fancy a look at http://github.com/wrboyce/erb, my (early days) Erlang/OTP IRC Bot Framework.
[...] Jones, one of the co-founders of Last.fm, has a writeup describing how Last.fm uses IRC. I’d really like to see IRC used for more stuff like this, including communication in [...]
Instead of Java, I use ii for that. <500 lines of C code transforms your file system in IRC channels.
We do something similar at Automattic. Private IRC network, ssl, password, protected. Bots for logging, Bots for announcing rss updates, trac changes, svn commits, deploys, nagios messages, etc…
Hah – at YellowBot we do the same.
[...] by how_gauche to programming [link] [114 comments] Go to Source Share and [...]
[...] we use IRC at Last.fm | Richard Jones, Esq. By mohangk How we use IRC at Last.fm | Richard Jones, Esq. Brilliant use of an IRC bot for system notification Possibly related posts: (automatically [...]
here’s a mini-sized ruby eventmachine port of netcat: http://github.com/purzelrakete/mini/tree/master
[...] прочтения статьи о том, как в last.fm используют irc для логирования всего и вся тоже захотелось как-нибудь [...]
[...] © 2008 Richard Jones Оригинал (Английский): How we use IRC at Last.fm Перевод: © orangeudav . Лицензия: All code licensed under GPLv2 unless otherwise [...]
[...] for TeamCity from Lance Fisher. I decided to implement it because I’ve always been a big fan of how last.fm use IRC to do notifications and queries. Last.fm puts everything up on IRC there from Jira issues, builds, hosting alerts and [...]
At Tencent (maker of the Chinese IM software QQ) we use RTX (internally developped and commercially sold software) internally, which is like an IM system modeled after the organizational structure. Most often you just use your own team, but it is also very useful to find other people in other business units. There is also integration with Outlook, bug trackers and so on.
While a bit insecure for us (QuickFox) to use nc in order to pass messages to IRC, I wrote an eggdrop script that would instead poll individual files (like the `tail -f` command) for new data and either send the data to pre-configured targets (channels/names), parse the data to be suitable for IRC output and only then send, or execute the data as if it was an IRC command. More “handlers” for specific data in these files could be added on to that script in order to extend it.
This way, we can restrict access to these files so only authorized users could write to them, and write various shell scripts to perform maintenance tasks or other things and report their findings to these text files. This way, we are informed on what’s going on on the server as we chat.
IRC is a great communication medium and it’s easy enough to extend it to monitoring purposes as well as making it a medium that connects between other mediums (such as connecting SecondLife, Instant Messengers and even gaming servers together through IRC). It is simple enough to allow access from my PDA or even cellphone, allowing me to have real-time conversations with people on mediums (such as Second Life) one would never dream of available on such mobile devices!
[...] [upmod] [downmod] How we use IRC at Last.fm | Richard Jones, Esq. (metabrew.com) 1 points posted 4 months, 1 week ago by jeethu tags monitoring interesting bot [...]
Can anyone comment on how to setup an IRC server (and/or a specific IRC server) that can cope with a high volume of messages coming from irccat? I’ve found that sending a high-enough volume of log data to a vanilla IRC server will provoke the server to throttle that logging – more or less defeating the purpose of mixing instant messaging and logging in the first place. Surely someone has come across this issue and solved it before me!
ooh, prior art! :P
http://www.stonehenge.com/merlyn/PerlJournal/col09.html
[...] is £60/mo (~100$) IRC is an absolute must, I use IRC for community/administration (similar to last.fm). I also have a personal bouncer. I’ll also be running a couple of small low traffic websites [...]
Загляните к нам в интернет-магазин для парикмахеров, где Вы найдёте продукцию компании Babyliss
[...] http://github.com/RJ/irccat/ and http://www.metabrew.com/article/how-we-use-irc-at-lastfm/ [2] [...]
[...] Komunikatory internetowe zastąpiły poczciwego IRC-a, ale nie oznacza to, że technologia ta odeszła w niepamięć. Wciąż można znaleźć osoby korzystające z niej. Po IRC-a sięgają też firmy, nierzadko duże (przykład użycia w Last.fm). [...]
[...] Komunikatory internetowe zastąpiły poczciwego IRC-a, ale nie oznacza to, że technologia ta odeszła w niepamięć. Wciąż można znaleźć osoby korzystające z niej. Po IRC-a sięgają też firmy, nierzadko duże (przykład użycia w Last.fm). [...]
[...] Richard Jones | How we use IRC at Last.fm | Richard Jones, Esq. [...]
Всем приветики! Сегодня облазила все магазины компьютерные. Смотрела нотбуки. Почему так цены взыинтились ? Почему дешевых вообще нет ? Нашла только пару дешевеньких, баксов за 500 это в магазе laptopnn.ru и еще в похожих пару штук.
Как у вас в городе с этим ? Тоже все глухо? или цены снижаются по-тихоньку ? Вот просто думаю, брать не брать….
[...] tiempo en estos sitios. Para esto, se tomaron 19 redes sociales de las más populares, como Digg, Last.fm, Xanga, Bebo, Hi5 entre [...]