Bembel-B Blog

2008/03/21

Prevent Debian on Linksys NSLU2 Going Out of Memory

After switching from Unslung to Debian I constantly got various programs (mt-daapd, rtorrent, gcc) terminating, because they ran out of memory. Tweaking or even disabling the oom-killer kernel feature didn’t do the trick, so I took a look at the ulimits and found most of them set to unlimited. Setting some of the ulimit defaults like on Ubuntu Linux solved these problems.
Linksys NSLU2

Update

Thanks to Michael the solution to any oom problem is upgrading to Debian Unstable (aka. Lenny). My “solution” didn’t proof. See the comments for more detail.

After some experimenting, I now have two ulimit settings – stack size and max locked memory – pointed out, but am not sure if really both are mandatory. But it shouldn’t hurt either. These limits, being unlimited by default, now got set to sane values as follows.

You can set the ulimits for the current shell with ulimit -l 32 -s 8192. System wide defaults are defined in /etc/security/limits.conf e.g. by adding this:

#restrictions to avoid oom using ubuntu defaults
*       hard    memlock 32
*       hard    stack   8192

As mentioned before, I currently have oom-killer disabled. But I should try to enable it again, to benefit from this feature. This can be done with the sysctl tool and in the /etc/sysctl.conf file.

While I’m at it, that would be the output of the stock Debian settings:

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
max nice                        (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) unlimited
max rt priority                 (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

That’s the output for Ubuntu Gutsy:

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 8191
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 8191
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

ChangeLog

[2008-04-11: Better upgrade to Lenny.]

Advertisements

2008/03/19

Fixed-point Ogg Vorbis and Musepack Decoder Packages for Debian Etch ARM

I’m using Firefly Media Server on a Linksys NSLU2 running Debian Etch to stream my whole music collection to my Pinnacle SoundBridge HomeMusic. Many files in my collection are in Ogg Vorbis format and a few in Musepack too, which unfortunately aren’t natively supported by the SoundBridge. Luckily Firefly can transcode audio files on the fly e.g. using the ssc plug-in and any console application via shell scripts. But the problem with the Debian packages is that they don’t take into account the missing floating point unit (FPU) of the NSLU2 hardware, and that makes oggdec/ogg123 and mpc123 take ages – way below real-time – to transcode. Fixed point versions of the transcoders aren’t available in the Debian Etch package repositories even the sourcecode already exists, so I had to build them myself.
Linksys NSLU2
I first searched the Debian packages for fixed point versions.
I found the libvorbisidec package, which is the fixed point version of libvorbis also going by the name Tremor. The vorbis-tools is only available as normal floating-point version linked to libvorbisdec. For Musepack there are (floating-point) packages libmpcdec3 for the library and mpc123 for the application.
In the Firefly forums I found patches for vorbis-tools (providing the ogg123 and oggdec transcoders), so that I could build Tremor versions of the transcoder apps.
In the Musepack sources there’s already a define variable to build with fixed point math only. So I would just have to enable that and rebuild the library and transcoder.

To make a long story short, here are the Debian Etch ARM binary packages for fixed-point Ogg Vorbis and Musepack:

I will gladly provide the sources. Feel free to ask for them! You can download the Deb sources for libmpcdec and vorbis-tools further below.

Update

These are the integer Lenny ARM binaries. Libvorbisidec is available through the official repos.:

Update 2

Here are the Deb sources (.dsc, .diff.gz and .orig.tar.gz) to build your own binary Deb packages.

Performance now is very good on my 266 MHz NSLU2. From what I’ve seen in top, mpc123 and ogg123 stay below 50% CPU usage and decoding is done faster than real-time.

Only to show off how unbelievably cool I am ;), and in case somebody would like to reproduce these builds on another distribution, I’ll roughly describe the steps I’ve taken.

Build libvorbisidec

Pretty straight forward. As far as I remember I only had to rebuild the source package:

mkdir -p ~/debuild/libvorbisdec
cd ~/debuild/libvorbisdec
dget -x http://ftp.de.debian.org/debian/pool/main/libv/libvorbisidec/libvorbisidec_1.0.2+svn14261-1.dsc
cd libvorbisidec-1.0.2
debuild -rfakeroot -uc -us
cd ..
sudo dpkg -i libvorbisidec1_1.0.2+svn14261-1_arm.deb libvorbisidec-dev_1.0.2+svn14261-1_arm.deb

Patch and build vorbis-tools

This one was trickier. The patches found on the forum aren’t complete. The older one is blindly changing the endianess of the resulting audio data, thereby producing only noise on the little endian Debian I use. The newer one misses linking mpcdec to libvorbisidec.

mkdir -p ~/debuild/vorbis-tools
cd ~/debuild/vorbis-tools
dget -x http://ftp.de.debian.org/debian/pool/main/v/vorbis-tools/vorbis-tools_1.1.1-6.dsc
cp -pr vorbis-tools_1.1.1 vorbis-tools_1.1.1.wrk
cd vorbis-tools_1.1.1.wrk
[apply and manually fix the tremor patch]
[create tremor patch file against vorbis-tools_1.1.1/]
[copy patch file to vorbis-tools_1.1.1/debian/patches]
[add patch file at bottom of vorbis-tools_1.1.1/debian/patches/series]
[add libvorbisidec-dev dependency to vorbis-tools_1.1.1/debian/control and document Tremor versions in description]
[increase version and add changes in changelog via dch -i]
cd vorbis-tools_1.1.1
debuild -rfakeroot -uc -us
cd ..
dpkg -i vorbis-tools_1.1.1-7_arm.deb

Patch and build libmpcdec

Here I add a patch to enable the fixed-point define. Beware, the following steps are just written down from memory in a hurry. So expect flaws. :)

mkdir -p ~/debuild/libmpcdec
cd ~/debuild/libmpcdec
dget -x http://ftp.de.debian.org/debian/pool/main/libm/libmpcdec/libmpcdec_1.2.2-1.dsc
cp -pr libmpcdec-1.2.2 libmpcdec-1.2.2.wrk
vim libmpcdec-1.2.2.wrk/include/mpcdec/math.h
[uncomment #define MPC_FIXED_POINT]
mkdir libmpcdec-1.2.2.wrk/debian/patches
diff -Naur libmpcdec-1.2.2 libmpcdec-1.2.2.wrk > libmpcdec-1.2.2.wrk/debian/patches/enable_fixed_point.diff
cd libmpcdec-1.2.2.wrk
dch -i
[increases version in changelog. add some notes.]
debuild -rfakeroot -uc -us
cd ..
dpkg -i libmpcdec3_1.2.2-2_arm.deb libmpcdec-dev_1.2.2-2_arm.deb

Build mpc123

This one has only to be rebuilt with the fixed-point libmpcdec version above installed.

mkdir -p ~/debuild/mpc123
cd ~/debuild/mpc123
dget -x http://ftp.de.debian.org/debian/pool/main/m/mpc123/mpc123_0.2.1-1.dsc
cd mpc123-0.2.2
debuild -rfakeroot -uc -us
cd ..
dpkg -i mpc123_0.2.2-1_arm.deb

I haven’t found a fast enough way to transcode Monkey’s Audio (aka. APE) files yet, but that’s mainly because I don’t have any (which haven’t been transcoded to mp3 by myself already). For your information: Building the Monkey’s Audio Codec (MAC) for Linux did work.
I think when looking for more low resource or fixed-point codecs, a good place to look for is the Rockbox Wiki. That’s what I used too.

To make this Firefly Transcoding post more complete, here’s my current version of the ssc script:

#!/bin/sh
#
# script to facilitate server-side transcoding of ogg files
#
#
# Usage: mt-daapd-ssc.sh   
#
# You may need to fix these paths:
#

WAVSTREAMER=/opt/mt-daapd-ssc/wavstreamer
OGGDEC=oggdec
OGG123=ogg123
FLAC=flac
MPC=mpc123
FFMPEG=/opt/mt-daapd-ssc/ffmpeg-quiet.sh
MAC=mac

ape_file() {
    $MAC "$FILE" - -d | $WAVSTREAMER -o $OFFSET $FORGELEN
}

ffmpeg_file() {
    $FFMPEG -i "$FILE" -acodec pcm_s16le -f wav - | $WAVSTREAMER -o $OFFSET $FORGELEN
}

mpc_file() {
    $MPC --quiet --wav - "$FILE" | $WAVSTREAMER -o $OFFSET $FORGELEN
}

ogg_file() {
#    $OGGDEC --quiet -o - "$FILE" | $WAVSTREAMER -o $OFFSET $FORGELEN
    $OGG123 -q -d wav -f - "$FILE" | $WAVSTREAMER -o $OFFSET $FORGELEN
}

flac_file() {
    $FLAC --silent --decode --stdout "$FILE" | $WAVSTREAMER -o $OFFSET $FORGELEN
}

FILE=$1
OFFSET=${2:-0}

# this is nonsense!
if [ "$3" != "" ]; then
  FORGELEN="-l ${3%.*}"
fi

case "$1" in
  *.[sS][hH][nN])
  ape_file
  ;;
  *.[sS][hH][nN])
  ffmpeg_file
  ;;
  *.[mM][pP][cC])
  mpc_file
  ;;
  *.[oO][gG][gG])
  ogg_file
  ;;
  *.[fF][lL][aA][cC])
  flac_file
  ;;
  *)
# here you could cat a generic "error" wav...
# cat /path/to/error.wav
  ;;
esac

Oh, FYI I’m using mt-daapd 0.2.4+r1376-1 from the Debian repo. I got the wavstreamer binary and the ssc script example by extracting them from a newer Debian package. The Firefly Wiki provides more information on transcoding implementation.

ChangeLog

[2008-04-11: Add Lenny Debs.]
[2008-06-06: Add Deb sources for libmpcdec and vorbis-tools.]
[2009-04-26: Fix box.net links.]

2008/03/13

ScummVM v0.11.1 for GP2X

The ScummVM binaries I found in the web aren’t up to date (v0.10) and therefore new games like Elvira series are missing. (v0.11). I already made some GP2X build of ScummVM (SVN) before. The current SVN didn’t run, so I built the latest stable version.ScummVM Logo

It’s built with the open2x toolchain against their standard SDL and the MAD (mp3), Tremor (Vorbis), and libmpeg2 (mpeg2) libraries. What’s missing is fluidsynth and FLAC, but those were also missing in the other GP2X builds AFAIK.

Download the build here: scummvm-gp2x-0.11.1.zip

One bug that persists in my builds is the stereo sound in games switches the balance to the right channel after a few seconds. Otherwise this build seems fine. I guess it has to do with the SDL_mixer in the open2x libraries.

ChangeLog

[2008-03-19: Categories for the post]
[2009-04-26: Fix box.net links.]

2008/03/11

Scrobbling Everywhere All the Time

I must confess I’ve become quite a Last.fm fanboy. :) So what would be more important than keeping track of as much music playing as possible. Scrobbling the plays of my PC audioplayer Amarok and Foobar2000 ain’t that spectacular, but feeding statistics of my mobile MP3 Player SanDisk Sansa e200 and my stand-alone player Pinnacle SoundBridge HomeMusic (licensed by Roku) I consider being more of that kind.Last.fm Social Music Revolution

Scrobbling Sansa e200 with Rockbox

Precondition is using the great alternative Firmware Rockbox. It already has the Audioscrobbler logging built in. To submit the logs I use the PC application QTScrobbler under Linux (and occasionally Windows). That’s very easy and convenient. Just be sure to set your Sansa’s clock somewhat correct.

Scrobbling SoundBridge with Firefly Mediaserver

To gain access to my whole music collection without having a PC running, I’m using the fine Linksys NSLU2 NAS running the Firefly Mediaserver (aka. mt-daapd) with a cheapo 160 GB USB HDD (Storage) and a 2 GB USB Flash Drive (OS) attached. I’ve had a working setup using the alternative NSLU2 firmware Unslung, but soon switched to Debian ARM, for its greater versatility and more straight forward configuration.

I’ll write more detailed posts on the NSLU2 soon, especially regarding Firefly and fixed-point Transcoding and Last.fm Radio. But for now a quick overview on the setup, which should be possible on other platforms and for any streaming client too.

I obtained the Firefly Mediaserver prebuilt from the Firefly website. Installation is quite easy and well documented.

Submission to Last.fm is done by the Python application Lastfmsubmitd. As the name suggests it’s a daemon permanently waiting for data to be submitted. That data is gathered from text files placed e.g. in /var/spool/lastfm. Under Unslung I had to manually install it from source (python setup.py install), and for Debian it’s in the apt repo (but I built a deb package of the recent version found in Debian unstable).

Creating the data files is done periodically by a shell script based on what I found in the Firefly Forum. It’s run every 5 minutes by cron and queries the “last played field” of Firefly’s collection database and outputs results to Lastfmsubmitd’s spool directory.
That’s my current shell script (converting GMT+1 timestamps to UTC by substracting 3600 seconds):

#!/bin/bash

# fetch newly played songs from fireflydb and write
# into lastfmsubmitd readable format

# config
SQLITE=sqlite3
DATABASE=/var/cache/mt-daapd/songs3.db
LASTFILE=/var/cache/mt-daapd/lastfmsubmit.date
DBLSFILE=/var/cache/mt-daapd/lastfmsubmit.ls
TMPDIR=/tmp
SPOOLDIR=/var/spool/lastfm


# get last run time
if [ -e "$LASTFILE" ]
then
  . "$LASTFILE"
else
  LASTRUN=0
fi

# get last database file date
if [ -e "$DBLSFILE" ]
then       
  . "$DBLSFILE"
else     
  DBLSRUN=
fi

# exit when database file unchanged
DBLSNOW=`ls -l "$DATABASE"`
if [ "$DBLSRUN" == "$DBLSNOW" ]
then
  exit
fi

# log file date
echo "DBLSRUN=\"$DBLSNOW\"" > "$DBLSFILE"

# query database
OUTFILE=$(mktemp "$TMPDIR"/mt-daapd-XXXXXXXX)
"$SQLITE" "$DATABASE" 'SELECT artist,album,title,track,song_length,time_played FROM songs where time_played > '"$LASTRUN"' ORDER BY time_played ASC;' | gawk -F '|' '{ printf "---\nartist: \"%s\"\nalbum: \"%s\"\ntitle: \"%s\"\ntrack: %s\nlength: %d\ntime: !timestamp %s\n",$1,$2,$3,$4,$5/1000,strftime("%Y-%m-%d %T",$6-3600) }' > "$OUTFILE"

# place non-zero result into spool, else drop file
if [ -s "$OUTFILE" ]
then
  chmod 664 "$OUTFILE"
  mv "$OUTFILE" "$SPOOLDIR"
else
  rm "$OUTFILE"
fi

# log query date
echo "LASTRUN="`date +%s` > "$LASTFILE"

Downside of this solution is, Firefly will only consider a track as played, if it has been completely and continuously been played. So skipping or pausing a track will cause it to not be submitted.
Also there’s no separation between Podcasts and the rest of my music collection. What I haven’t tried yet, is the behaviour when playing web radio via Firefly playlists, as I do all radio streaming directly through the SoundBridge user interface.
To iron out the downsides using the same approach, the first one would need changes to the Firefly code I guess, the others could probably be fixed modifying the shell script.

2006/10/15

Setting Up a Linux Cross Compiling Environment for GP2X with Gamepark Holdings’ SDK gp2xsdk

I recently bought myself the Gamepark Holdings GP2X a Linux driven handheld entertainment console. Arcade and gaming console emulators etc. work quite nicely on this machine. Also the potential is large, as it’s an all open architecture.

GPH GP2X

So I wanted to try a little development for this fine unit esp. with some C brand and SDL. As the GP2X has two ARM CPUs and compiling directly on the unit would be darn slow one would have to setup a cross compiling environment. You could do that on Windows or Linux or whatever, but I chose Linux of course. I use Fedora Core 5 on an x86 architecture.

Nota Bene

There are several ways to set up such a cross compiling environment and sadly the information on the nonetheless very useful GP2X Wiki is very sparse and outdated. As I’m totally unexperienced in cross compiling and only got basic (i.e. little not the “programming language” BASIC ;) knowledge of software development under Linux, I wanted to take a simple and mostly “official” approach with the SDK provided by Gamepark Holdings (GPH) released 06-07-2006 to get the compiler toolchain and the most important libraries.

Also there’s the problem that alternatives similar to my approach (section A and B in the aforementioned Wiki page), like ooPo’s build environment and the installer script by gamkiller, are not found on their authors websites anymore, are outdated, contain strange looking programm versions and the GPH SDK release is newer. So these alternatives didn’t seem very good to me. However these environments will be packed with more libraries than the GPH SDK.

Remind: This procedure works for me. There are very likely some errors or things not done the best way. Check back for changes!

One final foreword: To be able to run the programms built for the GP2X on your PC, you would have to setup some VM like Scratchbox. Scratchbox also contains a cross compiling environment. But this seemed to complicated for a first try, but you might want take a look at that!

Prepare the SDK Installation

I assume you got the basic development tools (gcc, g++ etc.) installed on your system.

Download the GPH SDK from the GPH website. Currently the URL is http://dev.gp2x.com/sdk/gp2xsdk_linux.tar.gz.
Extract it somewhere on your system.

Now edit the extracted buildgp2xtools.sh and change the installation destination defined by export PREFIX="/some/where" to your liking. Beware that this destination is hard coded; you cannot move the SDK afterwards without making it disfunctional. This destination must be writable for the user executing the build script in the next step.
For the path set there I will use ${PREFIX} as a placeholder from now on. So be sure to exchange ${PREFIX} with e.g. /some/where.

Build the SDK

Now cd to the destination of the extracted gp2xsdk_linux.tar.gz and run the build script buildgp2xtools.sh.

Additional files will be downloaded and then compilation starts. This all takes about 45 minutes on my Athlon XP 2500+ and needs about 800 MB disk space IIRC.

Everything has to go through without errors. If the build script aborts with an error, you’re maybe missing some build tools or other dependencies. Or ran out of disk space. ;) Investigate in this case!

All files will be put to the destination defined via PREFIX in buildgp2xtools.sh.

Install the Libraries

Extract gp2x-library.tar.gz contained in gp2xsdk_linux.tar.gz into ${PREFIX}/. That’s it for now.

Clean Up the SDK Build and Installer

The files downloaded by the build script can be deleted. They’re all in ${PREFIX}/files.

The build directory of the SDK can probably be deleted, as they were installed into ${PREFIX}/. They’re all in ${PREFIX}/build.

The extracted sources to build the SDK can probably also be deleted I guess. They’re all in ${PREFIX}/src. Not sure about this though. Maybe better do a backup of this directory before removing.

The files exctracted from gp2xsdk_linux.tar.gz can obviously be deleted.

Fix the Libraries and sdl-config Scripts

Some of the libraries’ files (*.la) from gp2x-library.tar.gz contain wrong path names. These have to be fixed. I adapted this from the install script by gamkiller. You must cd into ${PREFIX} and then run the following commands, with the sed substitution modified to your PREFIX path. Mine in this command is /home/scheff/wrk/gp2x/gph_sdk/gp2xsdk/Tools, a bit looong, eh. Note that the paths slashes have to be escaped (/ becomes \/).

for a in lib/*.la; do
cp -p "$a" "$a.orig"
sed -e 's/\/usr\/local\/gp2x-library\/image/\/home\/scheff\/wrk\/gp2x\/gph_sdk\/gp2xsdk\/Tools/g' "$a.orig" > "$a"
done

This will make a backup of all *.la files in ${PREFIX}/lib and change the default paths to yours.

The sdl-config also contains wrong paths. Strangely they differ from the default path found in the libraries above. Also the sdl-config is extisting twice (${PREFIX}/bin/sdl-config and ${PREFIX}/bin/arm-linux-sdl-config) with different contents. This shouldn’t be the case I think. So we will fix one sdl-config and create a symlink for the duplicate.

You could correct the path in the file easily with an editor. Its the prefix variable at the beginning of the file.

But as I wrote it and it looks soo ubercool, here’s a sed command. Again be sure to have cd’ed to your PREFIX and set the right path in the sed substitution statement:

a="bin/arm-linux-sdl-config"
cp -p "$a" "$a.orig"
sed -e 's/\/hd\/nk\/SDL\/image\//\/home\/scheff\/wrk\/gp2x\/gph_sdk\/gp2xsdk\/Tools/g' "$a.orig" > "$a"

Now for the symlinking for the dupe sdl-config:

pushd bin
rm sdl-config
ln -s arm-linux-sdl-config sdl-config
popd

Use the SDK

Finally you are now able to use the cross compiling environment! Congratulations and celebrations! :]

For the ease of use add the ${PREFIX}/bin to your PATH environment variable like this:

export PATH=${PREFIX}/bin:$PATH

A simple Hello World C++ application should now compile and run for your GP2X ARM CPU compiling with arm-gp2x-linux-g++.

Haven’t yet tried some C++ Hello World to stdout, but guess it’ll run fine in the console (e.g. sterm). My first attempt was the SDL Hello Pixel example, which I will cover in my next posting in this Blog. So much beforehand: it works! :)

Thanks and Credits

A thousand thanks and credits go to:
all the Open Source developers making this possible
GPH for the SDK
wiki.gp2x.org for providing the information
gamkiller for the installer script
some unknown people’s postings found via search engines
my mom and dad – and you for your interest in all this! :p

“Good night and good luck!”

ChangeLog

[070429 Add photo]

Blog at WordPress.com.