Blog entries from KDE SoC students and mentors

Amarok & NPR :: 13 Years of News Media Now Available at Your Fingertips

ramblurr — Thu, 06 Nov 2008 05:21:21 +0100

Earlier this summer I had noticed that National Public Radio (NPR) launched a brand new open API based on open source technologies. My initial reaction was at best skeptical. I assumed any sort of “API” released by a major media outlet would turn out to be nothing more than a few customizable RSS feeds. If the company was particularly progressive the RSS feeds might include full articles, rather than the neutered one-sentence teasers you find in all the big name’s syndicated content.
I couldn’t have been more mistaken. NPR’s API is no small potatoes. Just take a look at the comprehensive Query Generator to get an inkling of the types of complex queries you can create. Looking at the Query Generator also sheds some light on the content you can retrieve using the API. The API’s main page says the API exposes the entire NPR archive of content starting from the launch of the NPR website in 1995. Just how big is this archive? Over 250,000 stories including text, images, video, and audio!
This quote from the article announcing the API caught my eye immediately:

There were quite a few questions that we addressed when developing the API, but one thing that was not really in question was the need to open as much of our content as possible.1

This isn’t the first open media API. BBC was the first to offer a public open access API, however BBC’s API is restricted to the content from the past 7-days. Seven days! That’s nothing compared to the (approx.) 4748 days - and counting - that NPR’s API offers. NPR and the BBC are two large companies leading the technological shift towards open and free information.
But that’s only half the story.
After discovering this fantastic API I had to do something with it, and the new service architecture in Amarok 2 provided the perfect platform to build a NPR mashup. That was several months ago, and at the time the scripting API in Amarok was still being flesh out (Thanks to Peter). On Monday I noticed the BBC scriptable service Nikolaj had created for Amarok 2. I happened to have several hours of free time, so I cooked up a similar service for NPR:

You can get it at kde-apps or via the “Get More Scripts” button in Amarok 2’s Script Manager.
There is definitely room for improvement and in fact here are a few things I plan to do with it:

Display more than 20 stories under a category
Sort content by date
Support searching
Display the full articles, with images, in the context view

Major props and thanks go out to the entire NPR technical team and all the contributors who made API a reality.

NPR API is Live on NPR.org

Exporting contacts from Verizon to Gmail

ramblurr — Fri, 24 Oct 2008 01:10:30 +0200

I got a G1 today.
That deserves a post unto itself, but I wanted to share a solution an annoying issue regarding switching from Verizon to T-Mobile. Before I got the G1 I had a LG-VX9800 (yes, ancient, I know) with around 200 contacts. Obviously one of the first things I wanted to do when I got my G1 was transfer all my contacts from the LG to the G1. There are several ways to do this

Go to a Verizon store and pay $10 for data backup
Buy a USB Cable for the LG-VX9800, and use bitpim
Manually copy (type) your contacts into the G1/Gmail
Use my method

There was no way I was going to pay $10 for what should be a simple “Export,” so #1 was crossed out. I’m impatient and didn’t want to wait for a cable to be delivered and neither did I want to pay the money for one. There goes #2. I didn’t even consider #3; I just listed it for completeness sake.
That leaves my somewhat difficult and unreliable method. Basically, what I do is use Verizon’s “Backup Assistant” tool to send my contacts to Verizon’s website. Then I save the source of the “Print Contacts” page - because there is no export feature. With a little ruby I parse the file into Gmail’s CSV format and import the file via the Gmail contacts page.
Here’s a quick howto.

On your Verizon phone go to “Get Going -> Get a New App -> Backup Assistant” and install it for $1.50 a month.
Follow the prompts and backup your contacts.
Go to the Verizon backup website and sign in.
View your contacts and click the “Print Contacts” link. Save the source of this page to a file
Download this script and run it against the saved html file. Save the output in “contacts.csv”
Go to the Contacts page in Gmail and select Import, and upload “contacts.csv” Note: I suggest using the “add these imported contacts to” a new group feature. Because you will likely have to merge and cleanup the imported contacts.
Import and Enjoy

IMPORTANT: My script only grabs the following information from the Verizon contact list: Name, Email, Work Phone, Mobile Phone, and Home Phone.

Why Nepomuk could not fully replace (My)SQL in Amarok (yet)

DanielW — Sat, 11 Oct 2008 19:05:01 +0200

This is another post about collection backends in Amarok. The others posts explained the switch to MySQL embedded and tried to address some of the concerns. Jefferai also wrote a few things about Nepomuk in Amarok (and why it will not fully replace the sql based collections) which I want to comment on and add to.
There are user asking: “There is Nepomuk in KDE for storing meta data why the switch to MySQL embedded you could just use Nepomuk and drop sql collection all together.” With reasons like: “Nepomuk can do this and that better and one storage in KDE for everything is best.” They are not complete wrong, there are good reasons to use Nepomuk (data sharing between applications, system wide searching, using strigi(and taglib) for collection scanning at a central place for all applications once, interconnect the data with other data (like let Kmail store where I got that song from), and others)
But Jefferai is right when he says that Nepomuk can not replace the sql collection for everyone and every use case. At least not for the short and longer midterm (;-)).
Why? Not everyone will or could run Nepomuk for different reasons. The most important one: It is a quite complex big software in terms of memory usage (though I think that will improve in time and the amount of available memory in computers is growing fast) and CPU usage. Another issue is: At least in the short and midterm it it will be significant slower than the MySQL based collection. It is not useless slow but well sqlite isn’t useless slow either (faster than Nepomuk atm) and people complained. So: For slower older computers, lightweight media devices, mobile devices, big collections, java “haters” even a perfect done Nepomuk collection with a bug free Nepomuk (we are not there yet…) would be out of question.

I expect some of those issues to improve over time. (I hope we will see a faster C/C++ based storage backend for Nepomuk). But still Nepomuk is a new technology and there is not much experience with it yet amoung users, developers and important here “developing users” (those who create small scripts and apps to work on the sql data from amarok to do different things with that, mostly for their own usage).

Don’t get me wrong: I like Nepomuk and it possibilities for Amarok. It is not useless slow. In fact I am using it as my main collection backend in Amarok for months now. But it is in a similar state as KDE 4 in pre 4.0 days. You could use it but it not feature complete, it is not really stable it has its issues and there will be major changes to some of its parts. Well I hoped to have it (and the related Nepomuk service) in a better shape by now. Sorry for that, my time is very limited at the moment but work has not stopped fully and I am still planing to bring it into a useful shape.

A few words on the Nepomuk service: It will monitor Nepomuk/Strigi for new music and use taglib to read data Strigi can not read (yet?) to fill it. It then creates new music (tracks, artists and so on) resources in the Nepomuk storage. It should also inform interested applications about new music. Overall it aims to have always all music of the user in an easy to access way available for use in media players, music management applications and others.

That should make it easier (and faster!) for Amarok (and others) to use the data.

For example at the moment (ontology not final) to get a QStringList of all artists of the music on the computer in your application is as easy as (when using Strigi, Nepomuk and the unreleased service) :

QStringList artists;

QList<Nepomuk::Resource> artistsResources
    = Nepomuk::ResourceManager::allResourcesOfType( "http://purl.org/ontology/mo/MusicArtist" );

foreach( Nepomuk::Resource &artistResource, artistResources)

    artists << artistResource.label();

Why Nepomuk could not fully replace (My)SQL in Amarok (yet)

DanielW — Sat, 11 Oct 2008 19:05:01 +0200

That should make it easier (and faster!) for Amarok (and others) to use the data.

QStringList artists;

QList<Nepomuk::Resource> artistsResources
    = Nepomuk::ResourceManager::allResourcesOfType( "http://purl.org/ontology/mo/MusicArtist" );

foreach( Nepomuk::Resource &artistResource, artistResources)

    artists << artistResource.label();

The flexible Amarok 2 collection framework will allow both (Sql and Nepomuk) collection living together at users choice and maybe in a few years Nepomuk (with faster backend?) could become the default.

Gas measuring: Source, some charts, accuracy

DanielW — Sun, 28 Sep 2008 12:48:05 +0200

There was quite a lot of interest in my gas measuring project using a mobile phone. Thanks for that. For all who requested the source: I packed it together and wrote a small readme. You can download it now.
There were doubts about the accuracy and if it will work at all. It is running for over a week now without any crash or other problems. It measured a gas usage of 14.81 m³ while the gas meter is showing 13.85 m³. That is a difference of 6.9%. Not perfect, but i can live with that and can try to reset the “burning duration -> volume” factor to reflect the difference.
I also have first usage graphs:
Gas usage for September 2008
Day of month on x axis and used energy in kWh on y axis.
Does it mean a lot: No, not yet. Not enough days and also to much playing around with settings (temperature on thermostat in the kitchen, setting on the radiators).
One interesting fact is that it will always do exact 11 “burning sessions” every hour as long the temperature in the room with the thermostat has not reached the target temperature. But the duration of each of those sessions depends on the real needed energy from the radiators. Even if they do not request any energy it will do 11 burning starts but all of them very short.
That means: At least here is quite some room for optimization. I need to make sure that it only starts when at least one room requests energy. That is quite hard to achieve because there is only a thermostat for controlling it in one room.
Here a chart for an average day:
Gas usage graph for 2008/09/21
Hour of the day on x, burning duration in seconds on y
The two peaks are from the warm water heating. That starts about two times a day and takes about 250 seconds or 0.7 kWh each, means about 2 kWh (ca. 0.15 EUR ) a day for warm water.
Anyone still reading?

It’s getting cold: Measuring gas usage with S60 camera phone

DanielW — Sun, 21 Sep 2008 06:06:40 +0200

The last two weeks I haven’t done anything on KDE/Nepomuk/Amarok (Does that statement qualify for posting this on planetkde? ), but for a good reason: It’s getting cold outside.
So time to get back to an old project of me: I wanted to measure the gas usage of my gas heating in real-time and come up with the idea to use a mobile phone with a camera for that. The plan was to have the camera of a mobile phone looking on the burner and recognize if the flame is there or not. That should work under the assumption, that the burning duration is directly related to the used gas volume.
So I picked my old Nokia 6680 (Symbian S60 2nd FP2) (oh I am glad that you can now program it in Python using PyS60) and started work on a proof of concept.
To make it short: After not working on it for about 10 month, I manged to get it to work.
A small Python program calculates the amount of blue in a section of the camera view once every second and if it gets above some threshold it assumes the flame to be on and writes a record to a SQL DB on the phone. It also calculates the current gas meter reading and makes all that data available over Bluetooth on request.
Now a picture of the setup:
The gas heating with the phone looking on it (door open)
If the isn’t the coolest usage of a mobile phone And now a picture of what the phone is seeing when the flame is burning:

A am pretty satisfied with the accuracy of the measuring. The last day I had a measuring error of about 1,3%.
I also wrote a small Qt application to get the data from the phone (over a bluetooth spp connection) and visualize it. Also a screenshot:
Screenshot of gas monitor app
The next months it will be fun (OK, only energy saving addicts like me will consider this fun) to analyze the usage and see how it relates to the temperature. Hopefully it helps to save energy (and money ;-)). If someone is interested to build something similar, just ask me for the source. (But don’t expect it to be well written…)
Oh, and to somehow justify the syndication to the planet: I am back to work on the Nepomuk music service to hopefully get it into KDE 4.2 (the next post will be about that) so that it can get used in the first Amarok 2 feature release after KDE 4.2 (2.1 or 2.2)

GSoC Wrap Up

ramblurr — Thu, 18 Sep 2008 16:33:40 +0200

If there was a blogger award for “Most likely to make timely posts”, then in no possible world would I even be considered for the award. I could list some excuses that sound legit in my head, but the real reason I don’t feel motivated to post often (or on time) is because I’d rather spend that time in Google Reader reading everyone else’s exciting content (that was not sarcastic).
So, lets see… last time I posted I was en route to Akademy 2008 (the KDE developers conference). That was July 30th, now, a month and a half later I am back from Europe (which was amazing), GSoC is over (sad), and class has started (jury’s still out).
Current Status of the MP3tunes Amarok Service
The Good (Works)

Browsing & Streaming
Querymaker is as functional as possible with the current API
Manual Downloading from MP3tunes to Local Collection
Manual Uploading from any Collection to MP3tunes

The Bad (Doesn’t work)

AutoSync - 80%- The code is there, however there are problems with the daemon not receiving signals from the MP3tunes’ servers
MP3tunes playlists support - 0%- Never got started on this

The Ugly
There is one large issue I am still wrestling with. The details of the issue are complicated, but essentially it deals with the way Amarok handles remote tracks in playlists across sessions. Generally, remote tracks’ metadata isn’t cached by Amarok for use between multiple sessions. So, if you add an mp3tunes track to a playlist in Amarok, then restart Amarok and try to play that playlist the mp3tunes track is blank. Now, there exists a method for retrieving that metadata, however it was originally implemented synchronously. When you are retrieving metadata for any significant number of tracks (10+) synchronously, and each one of those retreivals is an HTTP Get request you end up blocking the GUI thread. At Akademy I hacked a way to do this asynchronously, but it is a really nasty hack. I’ve got code on my computer that implements this feature correctly, but it has the nasty habit of crashing Amarok every so often.
Between classes, marching band, homework, and other responsibilities I’m working on getting this ironed out and committed. Hopefully this will happen before the 2.0 release, because right now using MP3tunes in Amarok across sessions is slow and annoying.

Some statistics: sloccount

powerfox — Wed, 03 Sep 2008 12:50:00 +0200

Here is the statistics made with sloccount.
wc -l result I like more (~8500) :D
lcp script shows 8904 contributed lines (for generated patch).

Lines:
SLOC Directory SLOC-by-Language (Sorted)
1752 dvcs cpp=1751,sh=1
1220 git cpp=1219,sh=1
610 bazaar cpp=513,ansic=97
416 mercurial cpp=415,sh=1

Totals grouped by language (dominant language first):
cpp: 3898 (97.50%)
ansic: 97 (2.43%)
sh: 3 (0.08%)

FileCount:
#Files Directory #Files-by-Language (Sorted)
23 dvcs cpp=22,sh=1
13 git cpp=12,sh=1
11 bazaar cpp=10,ansic=1
10 mercurial cpp=9,sh=1

Totals grouped by language (dominant language first):
cpp: 53 (92.98%)
sh: 3 (5.26%)
ansic: 1 (1.75%)

Total Number of Files = 57
Total Number of Source Code Files = 57

GSoC is over

powerfox — Wed, 20 Aug 2008 00:01:00 +0200

Unfortunately GSoC has finished. It was a wonderful adventure: I got new friends, wrote some code (~8350 lines) and learnt new interesting things. I can say that after today’s commit to my git repo (with a lot of changes to yesterday’s commit, which is evaluted) I implemented all things I promised. But some things still require a lot of love.
I want to thank all people helped me during this summer:
Alexander Dymo(adymo, KDevelop) — My mentor who is strong both in GUI and Git.
Andreas Pakulat(apaku, KDevelop) — The man who can help with any part of KDevelop(or maybe even ith whole KDE && Qt).
Shawn O. Pearce (spearce, Git) — A man who is not in KDE, but who contacted my mentor and me to suggest his help.
Marco Costalba — QGit author, explained a lot of code from QGit.
Paul Mackerras — Gitk author, explained some basic algorithm (building rev history).
All guys from different IRC channels, mailinglist.
And of course Google for the amazing Open Source Program: Google summer of Code!

GSoC is over

powerfox — Wed, 20 Aug 2008 00:01:00 +0200

Unfortunately GSoC has finished. It was a wonderful adventure: I got new friends, wrote some code (~8350 lines counted by “wc -l”, so it includes all crap) and learnt new interesting things. I can say that after today’s commit to my git repo (with a lot of changes to yesterday’s commit, which is evaluted) I implemented all things I promised. But some things still require a lot of love.
I want to thank all people helped me during this summer:
Alexander Dymo(adymo, KDevelop) — My mentor who is strong both in GUI and Git.
Andreas Pakulat(apaku, KDevelop) — The man who can help with any part of KDevelop(or maybe even ith whole KDE && Qt).
Shawn O. Pearce (spearce, Git) — A man who is not in KDE, but who contacted my mentor and me to suggest his help.
Marco Costalba — QGit author, explained a lot of code from QGit.
Paul Mackerras — Gitk author, explained some basic algorithm (building rev history).
All guys from different IRC channels, mailinglist.
And of course Google for the amazing Open Source Program: Google summer of Code!