June 2016 downtime, what happened?

In June 2016, was periodically unavailable for several hours at a time, and then eventually offline for six days. Here’s what happened.

This post is written by Matt Lee, founder of

So we’re back — thankfully. The last few days have been tricky and worrying for

To give you an idea of our runs, day to day, let me talk a little about our infrastructure: runs on the Bytemark Cloud (aka BigV) — we have four servers for, two web servers, a load balancer and a database server. The problem was our database server.

As of right now (June 30th, 13:20pm Central Time) has 153330128 of your scrobbles — this represents about 120Gb of PostgreSQL data. Put simply, we ran out of disk space and PostgreSQL shut down.

The first thing we did was shut the server down, add more space and bring it back up. PostgreSQL wouldn’t start.

Next, I made a complete disk backup of all of the PostgreSQL files, and went to look at our backups on And they were broken — because of our disk space issues, we’d been failing to make a backup file, and our backup scripts had been backing up a 0 byte file. Not good.

Clint Adams, one of the developers had a look at the server, but as he was in Cape Town for DebConf, was unable to figure out the problem. I proceeded to follow some online tutorials for dealing with this, and eventually got PostgreSQL to start, but it was throwing errors about being unable to find things.

It turned out that the problem was that we had transaction wraparound — we’d done more than 2 billion transactions on the database (logins, scrobbles, etc) and the database had lost track of where we were. My reindexing efforts weren’t helping.

So, I restored the files a second time, and with help from RhodiumToad on IRC, used the built in diagnostic tools with PostgreSQL to understand the issue better, and to reset the transaction logs. Huge thanks to RhodiumToad! I found this tweet:

I was then able to upgrade PostgreSQL from version 9.1 to 9.4, put in a better backup script and tell our backup service to alert us if no backups happen for 2 days.

I was finally ready to bring the site back up, but first… I had a private look at the stats of the site: 153184145 scrobbles. That number is now 153330314 — an increase of 146169 — so while you’ve been busy listening to music, we’ve been busy learning about databases and the importance of proper backups.

It’s clear to me that we need to do a lot more on — I’m sorry we let things get this bad. I’m going to spend a few hours later this week looking at the most obvious bugs and improvements we can make to the site, and start working on those. If you have your own suggestions, please take a moment to register for our GitLab site and report the issue for us.

Follow on Twitter at @librefm or follow @mattl and @robmyers and see how Joan of Arc felt.


Filed under Uncategorized

This Giving Tuesday, support freedom

Today is Giving Tuesday in the US and in many other western countries too.

I thought I’d write a little bit about some organizations worth supporting today.

  • Free Software Foundation — the stewards of the GNU License Suite, including the GNU Affero GPL that we use for The FSF is also an active part of our community, as they provide support for our GNU FM project.
  • Creative Commons — the stewards of the licenses we use for everything from the data on to the vast majority of the music we promote. CC’s two free culture licenses and work on the public domain are great tools for cultural freedom.
    (Disclaimer: Rob Myers and Matt Lee both work for Creative Commons now.)
  • Software Freedom Conservancy — home to many pieces of key free software, including Outreachy and supporters of

And some other groups in our space worth donating to as well:


Leave a comment

Filed under Uncategorized

133 million songs and counting

Today, we’re talking about another milestone for — 133 million song listens. Previously, in March 2013 we talked about 72 million listens, and a userbase of approximately 100,000 users. Today, some two and a half years later we’re at almost double that, with just over 200,000 users and 133 million song listens (with some 19.5 million of those being unique too!)

What do you do with data that big? We’re not sure. We’re certainly committed to making something cool with it, and we’d like to add better tools to allow people to export their own listening data and play around with it, graph it, etc.

When I talk about in modern terms, I think of each of those song listens as a tweet, or in the language of 2009, a dent — people are now actively sharing their listening habits in different ways, conversing about and actively publishing their listening habits to social media in real time. It’s a nice way to think about it too — users have 133 million published song listens — which is pretty amazing!

I have a lot of ideas for how to make better and I hope soon we’ll have some time to get them done.

It’s amazing to think what we can do next when we get a better interface to this data, and I think we have some interesting ideas on how to do that next🙂

Leave a comment

Filed under Uncategorized

More news — server move, new design, kittens.

This month sees a massive update to’s infrastructure thanks to our good friends at Bytemark.

Bytemark have generously donated us a ton of resources for including a new database server with super fast solid state disks, and a pair of load balanced high speed web servers for the site.

What this means for everyone else is that things are about to get a lot faster.

This does however mean that there will be some downtime for the site, during which time we won’t be accepting scrobbles. If your client is any good, it’ll cache those scrobbles for you, and when we come back online, your listening history will be sent over. Traditionally, coming back online after a little bit of downtime means we get a TON of scrobbles submitted to us, which slows the site down a bunch, so it’ll be interesting to see how well it works on the new setup.

Huge thanks to Matthew, Peter and Tim over at Bytemark for making this happen and their continued support.

In addition, we hope to launch the new design of the site at the same time.’s own Matt Lee and Rob Myers will be on-hand during the migration to answer any of your questions — grab us on Twitter or GNU social @librefm (we’re on if you use GNU social)

Oh, and we have a donations page where we accept all kinds of donations and high-fives and even Bitcoin (which we don’t know how to really use yet, so suggestions are welcome on that too)

Kittens? Here’s a GIF of a cat eating a watermelon.


Leave a comment

Filed under Uncategorized

Support the Free Software Foundation when buying stuff on Amazon

So, this is pretty fantastic, especially if like me you buy a ton of stuff from Amazon Prime.

  1. Go to and sign in with your Amazon account.
  2. Search for “Free Software Foundation” in the list of charities, and pick the one in Boston, MA.
  3. Buy stuff on Amazon.

0.5% of all your purchases go to the FSF.

Screenshot from 2013-12-19 16:08:39

A great way to support the FSF and — the FSF is the host organization for GNU, and runs on the GNU FM we wrote, which is part of GNU. Phew.

PS. If you’re not following @librefm on GNU social (our other project) or @librefm on Twitter, you should do so.

PPS. Yes, we know Amazon has a lot of issues like Kindle DRM and doesn’t treat small publishers (like the FSF’s GNU Press) very well, but it’s a great place to buy large bags of candy, jars of Red Vines (Twizzlers suck!) and music by some of the artists on, which is another way you can support artists.

Leave a comment

Filed under Uncategorized news for September 2013

This weekend, there’s a hackathon for the GNU FM
software that powers If you’re interested in
improving the site, please join us on IRC or in person
at MIT in Cambridge, Massachusetts.

We’ll be focusing on the new look and feel for the site
this weekend, but now is a good time to get involved if
you have any PHP skills too, as we have plenty of other
interesting things to work on as well.

Five years ago, I had the pleasure to work with the
comedian and author Stephen Fry on a short film about
GNU, which you can watch at:

I’ll be at the event in Cambridge, so stop by and say

If you can join us in person, check out
or if you’re attending virtually,
we’ll be in on Freenode. Join us using a
standard IRC client (such as XChat) or via your web

# GNU social news

Our sister project, GNU social (previously known as
StatusNet) will also be in attendance this weekend, and
I’ll be running hackathons for both projects from a
single room.

At GNU 30th, we’ll be launching — a GNU
social site for the GNU Project, as well as a couple
other GNU social sites, including one open to the public.

As we roll it out slowly, more details and your
exclusive invite code will be sent to you via email.

# And in other news:

Bytemark Hosting in the UK has stepped up as a new host for and we’ll be moving the site over to their new infrastructure in the coming weeks, including putting our mammoth database onto a series of very fast solid-state drives, which will make things faster for everyone. Thanks to Tim and Matthew at Bytemark for making everything awesome.

Donations for are welcome in a variety of ways,

incluing Flattr and Bitcoins, as well as PayPal:

You can follow me on Twitter (and soon GNU social) @mattl and GNU social are also on Twitter: @librefm @gnusocial

Leave a comment

Filed under Uncategorized

Use and GNU FM and help defeat surveillance

Right now, we’re working on a brand new look and feel for all GNU FM sites, based on the latest release of Bootstrap (we promise it’ll be less purple) and I thought I’d take a few minutes to share some ideas I’ve had recently on the subject of privacy and surveillance.

As you hopefully know if you use, we do not log your personally identifiable information, such as your IP address, screen resolution, etc. Almost every other website does this in some way, either with a program such as Google Analytics, or with their server logs.

Here’s exactly what we store on the site right now: - - [25/Aug/2013:06:26:02 -0700] "GET /themes/librefm/img/player/seek-backward.png HTTP/1.1" 304 188 "-" "Mozilla/5.0 (compatible; YandexImages/3.0; +" - - [25/Aug/2013:06:25:57 -0700] "GET /2.0/?method=track.gettoptags&artist=Tryad&track=Lovely&format=json HTTP/1.1" 200 3444 "" "Mozilla/5.0 (X11; Linux x86_64; rv:23.0) Gecko/20100101 Firefox/23.0 Iceweasel/23.0.1"

In these results, we can see that a search engine indexed one of our images, and that another user tried to find the top tracks by Tryad, but we don’t know which user is doing this, or where they are located.

This is not true of sites like, and especially not true of Facebook. Facebook apps work with sites such as
Rdio, Spotify and others to show up to the minute live listening information from other sites, right in the users’ Facebook action stream.

For maximum privacy, we recommend installing your own GNU FM server. We have already demonstrated limited federation abilities at FOSDEM last year, and have recently green lit our work on the GNU social project which will give us a strong, federated social presence, and technology we can piggy back onto.

With the 30th anniversary of the GNU project coming, now is an important time for all GNU projects to work together on providing a real alternative to sites that users cannot control.

Leave a comment

Filed under Uncategorized