The YouTube Conspiracy in Abby/CClive

There is a command line utility available on all Ubuntu derivatives called cclive. If you install it (sudo apt-get install cclive), you can give it a YouTube URL and it will download the video on it. I love using it for backup purposes - I upload a video from the camera, perform all my changes on YouTube, and then cclive the outcome for posterity. Just in case YouTube "forgets" about my latest cam ride.

There is also a GUI for cclive, called Abby. Abby is more than just a frontend for cclive, though, it also helps with pages that have multiple videos on them - playlists or RSS feeds. Abby is hosted on googlecode and written in QT/C++.

I started surfing, so I decided to scour YouTube for surf instruction videos. There is a set of 12 introductory videos by two Australian pros on there, so I decided to download them to take them to the beach. My N900 gladly displays YouTube videos on its gorgeous screen. Unfortunately, though, the T-Mobile coverage is fairly bad, so a downloaded video was the only real option.

Turns out (as I mentioned in a separate post) that cclive for lucid doesn't work with YouTube. Like, at all. It spits out an error (404) and dies. It's very embarrassing, really. Even more embarrassingly, the Ubuntu folks have the fix, but won't release it in lucid. They would just have to issue an update/bug fix, especially important since this is a long-term release. But no, the fix is in maverick (the next version, due out in October).

I decided to go for it and to install the maverick version from packages.ubuntu.com. You need to download a separate library, too (libquvi). You just go to the link, hit "Download i386," save the package wherever you like, do the same with libquvi0 (accessible from the same page). Then, with the two .deb files, you issue:

sudo dpkg -i *.deb

There is going to be a ton of blah-blah, and then you have your new version of cclive that should be working.

Well, when I fired up cclive with a link to the videos, it downloaded them just fine. Great! now I could use ffmpeg to transform the videos into something that the N900 would like - adjusting the resolution to the screen size?

Then I took up abby. It is nice in that, when you feed it a YouTube page, it fetches the titles of the videos and saves them as such, instead of the ugly hashed name generated by cclive. When I tried to do that, though, it found the videos, found the titles, but told me that www.youtube.com was an unsupported site.

Huh?

The documentation says explicitly that YouTube is supported. As a matter of fact, it's the only example it cites! I thought about it and started thinking this was a conspiracy: the code was hosted on Google Code - the company that does no evil had decided that you should be allowed to download any videos, except those on YouTube, which it owns.

You can't fool me! If it's in the Ubuntu repo, there must be a source package with it! I quickly typed:

apt-get source abby

and a tiny little blah-blah later I had a directory with the source code. I was going to find the place where it says YouTube is an exception and stick it to the men!!!

Only, there was no such place. While YouTube was not covered, it was not explicitly excluded. Instead, as I read the source, I realized that the supported formats come from a call to cclive --hosts. I called cclive --hosts, and out came a short list of domains. The last one was youtube.com. There also was a google.com, which indicates that there is no ill intent on the part of Google.

The final hint came from a comment in the abby code:

            lst.removeLast(); // The note line.

(mainwnd.cpp, line 167)
Evidently, the previous version of cclive had a note line below the list of formats that the new version doesn't have. So. youtube.com was chopped off because it was the last one reported, not because it was the one that Google doesn't like.

Looking at the code, removing the last line is a stupid idea. The set of lines is distinguished by a tab (\t) which separates the domain from the format hints to use. If the line contains no tab, it's ignored anyway. So the correct solution to the problem would have been to just remove the offending line 167. That, though, would have required me to compile the software, which would have made it incompatible with apt-get. Grumble, grumble... My cclive was non-standard, so I wondered if there was something I could do there.

Turns out abby allows you to specify the name and location of the cclive executable. So I just wrote a script that passes all arguments down to cclive, except for the argument --hosts. That one it passes through, but adds a non-empty line at the end of the output.

The whole script is seven lines and finally got the abby code to work as expected!