Fixed-point Ogg Vorbis and Musepack Decoder Packages for Debian Etch ARM

I’m using Firefly Media Server on a Linksys NSLU2 running Debian Etch to stream my whole music collection to my Pinnacle SoundBridge HomeMusic. Many files in my collection are in Ogg Vorbis format and a few in Musepack too, which unfortunately aren’t natively supported by the SoundBridge. Luckily Firefly can transcode audio files on the fly e.g. using the ssc plug-in and any console application via shell scripts. But the problem with the Debian packages is that they don’t take into account the missing floating point unit (FPU) of the NSLU2 hardware, and that makes oggdec/ogg123 and mpc123 take ages – way below real-time – to transcode. Fixed point versions of the transcoders aren’t available in the Debian Etch package repositories even the sourcecode already exists, so I had to build them myself.

I first searched the Debian packages for fixed point versions.
I found the libvorbisidec package, which is the fixed point version of libvorbis also going by the name Tremor. The vorbis-tools is only available as normal floating-point version linked to libvorbisdec. For Musepack there are (floating-point) packages libmpcdec3 for the library and mpc123 for the application.
In the Firefly forums I found patches for vorbis-tools (providing the ogg123 and oggdec transcoders), so that I could build Tremor versions of the transcoder apps.
In the Musepack sources there’s already a define variable to build with fixed point math only. So I would just have to enable that and rebuild the library and transcoder.

To make a long story short, here are the Debian Etch ARM binary packages for fixed-point Ogg Vorbis and Musepack:

• libvorbisidec
• libvorbisidec-dev
• vorbis-tools
• libmpcdec
• libmpcdec-dev
• mpc123

I will gladly provide the sources. Feel free to ask for them! You can download the Deb sources for libmpcdec and vorbis-tools further below.

Update

These are the integer Lenny ARM binaries. Libvorbisidec is available through the official repos.:

• vorbis-tools
• libmpcdec
• libmpcdec-dev
• mpc123

Update 2

Here are the Deb sources (.dsc, .diff.gz and .orig.tar.gz) to build your own binary Deb packages.

• libmpcdec_1.2.2-2.debsrc.tar.gz
• vorbis-tools_1.2.0-2.debsrc.tar.gz

Performance now is very good on my 266 MHz NSLU2. From what I’ve seen in top, mpc123 and ogg123 stay below 50% CPU usage and decoding is done faster than real-time.

Only to show off how unbelievably cool I am ;), and in case somebody would like to reproduce these builds on another distribution, I’ll roughly describe the steps I’ve taken.

Build libvorbisidec

Pretty straight forward. As far as I remember I only had to rebuild the source package:

mkdir -p ~/debuild/libvorbisdec
cd ~/debuild/libvorbisdec
dget -x http://ftp.de.debian.org/debian/pool/main/libv/libvorbisidec/libvorbisidec_1.0.2+svn14261-1.dsc
cd libvorbisidec-1.0.2
debuild -rfakeroot -uc -us
cd ..
sudo dpkg -i libvorbisidec1_1.0.2+svn14261-1_arm.deb libvorbisidec-dev_1.0.2+svn14261-1_arm.deb

Patch and build vorbis-tools

This one was trickier. The patches found on the forum aren’t complete. The older one is blindly changing the endianess of the resulting audio data, thereby producing only noise on the little endian Debian I use. The newer one misses linking mpcdec to libvorbisidec.

mkdir -p ~/debuild/vorbis-tools
cd ~/debuild/vorbis-tools
dget -x http://ftp.de.debian.org/debian/pool/main/v/vorbis-tools/vorbis-tools_1.1.1-6.dsc
cp -pr vorbis-tools_1.1.1 vorbis-tools_1.1.1.wrk
cd vorbis-tools_1.1.1.wrk
[apply and manually fix the tremor patch]
[create tremor patch file against vorbis-tools_1.1.1/]
[copy patch file to vorbis-tools_1.1.1/debian/patches]
[add patch file at bottom of vorbis-tools_1.1.1/debian/patches/series]
[add libvorbisidec-dev dependency to vorbis-tools_1.1.1/debian/control and document Tremor versions in description]
[increase version and add changes in changelog via dch -i]
cd vorbis-tools_1.1.1
debuild -rfakeroot -uc -us
cd ..
dpkg -i vorbis-tools_1.1.1-7_arm.deb

Patch and build libmpcdec

Here I add a patch to enable the fixed-point define. Beware, the following steps are just written down from memory in a hurry. So expect flaws. :)

mkdir -p ~/debuild/libmpcdec
cd ~/debuild/libmpcdec
dget -x http://ftp.de.debian.org/debian/pool/main/libm/libmpcdec/libmpcdec_1.2.2-1.dsc
cp -pr libmpcdec-1.2.2 libmpcdec-1.2.2.wrk
vim libmpcdec-1.2.2.wrk/include/mpcdec/math.h
[uncomment #define MPC_FIXED_POINT]
mkdir libmpcdec-1.2.2.wrk/debian/patches
diff -Naur libmpcdec-1.2.2 libmpcdec-1.2.2.wrk > libmpcdec-1.2.2.wrk/debian/patches/enable_fixed_point.diff
cd libmpcdec-1.2.2.wrk
dch -i
[increases version in changelog. add some notes.]
debuild -rfakeroot -uc -us
cd ..
dpkg -i libmpcdec3_1.2.2-2_arm.deb libmpcdec-dev_1.2.2-2_arm.deb

Build mpc123

This one has only to be rebuilt with the fixed-point libmpcdec version above installed.

mkdir -p ~/debuild/mpc123
cd ~/debuild/mpc123
dget -x http://ftp.de.debian.org/debian/pool/main/m/mpc123/mpc123_0.2.1-1.dsc
cd mpc123-0.2.2
debuild -rfakeroot -uc -us
cd ..
dpkg -i mpc123_0.2.2-1_arm.deb

I haven’t found a fast enough way to transcode Monkey’s Audio (aka. APE) files yet, but that’s mainly because I don’t have any (which haven’t been transcoded to mp3 by myself already). For your information: Building the Monkey’s Audio Codec (MAC) for Linux did work.
I think when looking for more low resource or fixed-point codecs, a good place to look for is the Rockbox Wiki. That’s what I used too.

To make this Firefly Transcoding post more complete, here’s my current version of the ssc script:

#!/bin/sh
#
# script to facilitate server-side transcoding of ogg files
#
#
# Usage: mt-daapd-ssc.sh   
#
# You may need to fix these paths:
#

WAVSTREAMER=/opt/mt-daapd-ssc/wavstreamer
OGGDEC=oggdec
OGG123=ogg123
FLAC=flac
MPC=mpc123
FFMPEG=/opt/mt-daapd-ssc/ffmpeg-quiet.sh
MAC=mac

ape_file() {
    $MAC "$FILE" - -d | $WAVSTREAMER -o $OFFSET $FORGELEN
}

ffmpeg_file() {
    $FFMPEG -i "$FILE" -acodec pcm_s16le -f wav - | $WAVSTREAMER -o $OFFSET $FORGELEN
}

mpc_file() {
    $MPC --quiet --wav - "$FILE" | $WAVSTREAMER -o $OFFSET $FORGELEN
}

ogg_file() {
#    $OGGDEC --quiet -o - "$FILE" | $WAVSTREAMER -o $OFFSET $FORGELEN
    $OGG123 -q -d wav -f - "$FILE" | $WAVSTREAMER -o $OFFSET $FORGELEN
}

flac_file() {
    $FLAC --silent --decode --stdout "$FILE" | $WAVSTREAMER -o $OFFSET $FORGELEN
}

FILE=$1
OFFSET=${2:-0}

# this is nonsense!
if [ "$3" != "" ]; then
  FORGELEN="-l ${3%.*}"
fi

case "$1" in
  *.[sS][hH][nN])
  ape_file
  ;;
  *.[sS][hH][nN])
  ffmpeg_file
  ;;
  *.[mM][pP][cC])
  mpc_file
  ;;
  *.[oO][gG][gG])
  ogg_file
  ;;
  *.[fF][lL][aA][cC])
  flac_file
  ;;
  *)
# here you could cat a generic "error" wav...
# cat /path/to/error.wav
  ;;
esac

Oh, FYI I’m using mt-daapd 0.2.4+r1376-1 from the Debian repo. I got the wavstreamer binary and the ssc script example by extracting them from a newer Debian package. The Firefly Wiki provides more information on transcoding implementation.

ChangeLog

[2008-04-11: Add Lenny Debs.]
[2008-06-06: Add Deb sources for libmpcdec and vorbis-tools.]
[2009-04-26: Fix box.net links.]

Scrobbling Everywhere All the Time

I must confess I’ve become quite a Last.fm fanboy. :) So what would be more important than keeping track of as much music playing as possible. Scrobbling the plays of my PC audioplayer Amarok and Foobar2000 ain’t that spectacular, but feeding statistics of my mobile MP3 Player SanDisk Sansa e200 and my stand-alone player Pinnacle SoundBridge HomeMusic (licensed by Roku) I consider being more of that kind.

Scrobbling Sansa e200 with Rockbox

Precondition is using the great alternative Firmware Rockbox. It already has the Audioscrobbler logging built in. To submit the logs I use the PC application QTScrobbler under Linux (and occasionally Windows). That’s very easy and convenient. Just be sure to set your Sansa’s clock somewhat correct.

Scrobbling SoundBridge with Firefly Mediaserver

To gain access to my whole music collection without having a PC running, I’m using the fine Linksys NSLU2 NAS running the Firefly Mediaserver (aka. mt-daapd) with a cheapo 160 GB USB HDD (Storage) and a 2 GB USB Flash Drive (OS) attached. I’ve had a working setup using the alternative NSLU2 firmware Unslung, but soon switched to Debian ARM, for its greater versatility and more straight forward configuration.

I’ll write more detailed posts on the NSLU2 soon, especially regarding Firefly and fixed-point Transcoding and Last.fm Radio. But for now a quick overview on the setup, which should be possible on other platforms and for any streaming client too.

I obtained the Firefly Mediaserver prebuilt from the Firefly website. Installation is quite easy and well documented.

Submission to Last.fm is done by the Python application Lastfmsubmitd. As the name suggests it’s a daemon permanently waiting for data to be submitted. That data is gathered from text files placed e.g. in /var/spool/lastfm. Under Unslung I had to manually install it from source (python setup.py install), and for Debian it’s in the apt repo (but I built a deb package of the recent version found in Debian unstable).

Creating the data files is done periodically by a shell script based on what I found in the Firefly Forum. It’s run every 5 minutes by cron and queries the “last played field” of Firefly’s collection database and outputs results to Lastfmsubmitd’s spool directory.
That’s my current shell script (converting GMT+1 timestamps to UTC by substracting 3600 seconds):

#!/bin/bash

# fetch newly played songs from fireflydb and write
# into lastfmsubmitd readable format

# config
SQLITE=sqlite3
DATABASE=/var/cache/mt-daapd/songs3.db
LASTFILE=/var/cache/mt-daapd/lastfmsubmit.date
DBLSFILE=/var/cache/mt-daapd/lastfmsubmit.ls
TMPDIR=/tmp
SPOOLDIR=/var/spool/lastfm


# get last run time
if [ -e "$LASTFILE" ]
then
  . "$LASTFILE"
else
  LASTRUN=0
fi

# get last database file date
if [ -e "$DBLSFILE" ]
then       
  . "$DBLSFILE"
else     
  DBLSRUN=
fi

# exit when database file unchanged
DBLSNOW=`ls -l "$DATABASE"`
if [ "$DBLSRUN" == "$DBLSNOW" ]
then
  exit
fi

# log file date
echo "DBLSRUN=\"$DBLSNOW\"" > "$DBLSFILE"

# query database
OUTFILE=$(mktemp "$TMPDIR"/mt-daapd-XXXXXXXX)
"$SQLITE" "$DATABASE" 'SELECT artist,album,title,track,song_length,time_played FROM songs where time_played > '"$LASTRUN"' ORDER BY time_played ASC;' | gawk -F '|' '{ printf "---\nartist: \"%s\"\nalbum: \"%s\"\ntitle: \"%s\"\ntrack: %s\nlength: %d\ntime: !timestamp %s\n",$1,$2,$3,$4,$5/1000,strftime("%Y-%m-%d %T",$6-3600) }' > "$OUTFILE"

# place non-zero result into spool, else drop file
if [ -s "$OUTFILE" ]
then
  chmod 664 "$OUTFILE"
  mv "$OUTFILE" "$SPOOLDIR"
else
  rm "$OUTFILE"
fi

# log query date
echo "LASTRUN="`date +%s` > "$LASTFILE"

Downside of this solution is, Firefly will only consider a track as played, if it has been completely and continuously been played. So skipping or pausing a track will cause it to not be submitted.
Also there’s no separation between Podcasts and the rest of my music collection. What I haven’t tried yet, is the behaviour when playing web radio via Firefly playlists, as I do all radio streaming directly through the SoundBridge user interface.
To iron out the downsides using the same approach, the first one would need changes to the Firefly code I guess, the others could probably be fixed modifying the shell script.