Bembel-B Blog

2008/03/21

Prevent Debian on Linksys NSLU2 Going Out of Memory

After switching from Unslung to Debian I constantly got various programs (mt-daapd, rtorrent, gcc) terminating, because they ran out of memory. Tweaking or even disabling the oom-killer kernel feature didn’t do the trick, so I took a look at the ulimits and found most of them set to unlimited. Setting some of the ulimit defaults like on Ubuntu Linux solved these problems.
Linksys NSLU2

Update

Thanks to Michael the solution to any oom problem is upgrading to Debian Unstable (aka. Lenny). My “solution” didn’t proof. See the comments for more detail.

After some experimenting, I now have two ulimit settings – stack size and max locked memory – pointed out, but am not sure if really both are mandatory. But it shouldn’t hurt either. These limits, being unlimited by default, now got set to sane values as follows.

You can set the ulimits for the current shell with ulimit -l 32 -s 8192. System wide defaults are defined in /etc/security/limits.conf e.g. by adding this:

#restrictions to avoid oom using ubuntu defaults
*       hard    memlock 32
*       hard    stack   8192

As mentioned before, I currently have oom-killer disabled. But I should try to enable it again, to benefit from this feature. This can be done with the sysctl tool and in the /etc/sysctl.conf file.

While I’m at it, that would be the output of the stock Debian settings:

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
max nice                        (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) unlimited
max rt priority                 (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

That’s the output for Ubuntu Gutsy:

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 8191
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 8191
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

ChangeLog

[2008-04-11: Better upgrade to Lenny.]

Advertisements

7 Comments »

  1. Great article! I have the same problem with OOM when I switched from Unslung to Debian. I added the lines to /etc/security/limits.conf:
    #restrictions to avoid oom using ubuntu defaults
    * hard memlock 32
    * hard stack 8192

    but this didn’t do the trick for me. Rtorrent is still crashing and dmesg states the OOM issue. Can you be a little more specific on the tweaks I have to make on /etc/sysctl.conf file ?
    How much swap does your system use ?

    Comment by Michael — 2008/04/04 @ 10:19 | Reply

  2. Hey Michael,

    thanks for your reply. Funny enough I had these oom errors again a few days ago. I was trying to download bigger torrents as before, so that seems to be the cause. I can’t limit the resident memory usage via .rtorrent.rc, because that also limits the virtual memory (disk access) and results in downloaded chunks not being written to the disk.
    Here’s the current .rtorrent.rc I use on my Slug:

    min_peers = 10
    max_peers = 80
    min_peers_seed = 5
    max_peers_seed = 40
    max_uploads = 3
    upload_rate = 11
    directory = /mnt/HDDRIVE2GO/torrents
    session = /mnt/HDDRIVE2GO/torrents/.rtorrent_session
    schedule = watch_directory,5,5,load_start=/home/scheff/rtorrent-watch/*.torrent
    schedule = untied_directory,5,5,stop_untied=
    schedule = low_diskspace,5,60,close_low_diskspace=100M
    schedule = ratio,60,60,stop_on_ratio=120,125M,250
    port_range = 8661-8679
    port_random = yes
    max_memory_usage = 128M
    send_buffer_size = 128K
    receive_buffer_size = 1M
    max_open_files = 256
    max_open_sockets = 512
    hash_read_ahead = 5
    hash_interval = 10
    hash_max_tries = 3
    check_hash = no
    use_udp_trackers = yes
    session_lock = yes

    My Slug currently has about 24 MB swapped, that’s without rtorrent running but a couple of other services (mt-daapd, lighttpd with php5, lastfmproxy, kernel-nfs, etc.).

    What I was referring to with sysctl.conf are the settings relevant for oom-killer and the so called laptop-mode. On my system these kernel parameters are set by my modified init script kernel-param I found in David Härdeman’s article. It would certainly be more elegant to set these values in sysctl.conf though.
    This is my kernel-params files:

    #!/bin/sh

    PATH=/sbin:/bin

    # Max time to wait for writeout
    MAX_AGE=120
    CENT_AGE=$((100 * $MAX_AGE))
    # Max percent of mem to use for dirty pages
    DRATIO=10
    # Once we write, do so until this many percent of mem is still in use
    DBRATIO=1
    # Max ram overcommit percentage
    OOMRATIO=75
    # Overcommit policy
    OOMMEM=2

    case “$1” in
    start)
    echo -n 0 >> /proc/sys/vm/swappiness
    echo -n $MAX_AGE >> /proc/sys/vm/laptop_mode
    echo -n $CENT_AGE >> /proc/sys/vm/dirty_writeback_centisecs
    echo -n $CENT_AGE >> /proc/sys/vm/dirty_expire_centisecs
    echo -n $DRATIO >> /proc/sys/vm/dirty_ratio
    echo -n $DBRATIO >> /proc/sys/vm/dirty_background_ratio
    echo -n $OOMRATIO >> /proc/sys/vm/overcommit_ratio
    echo -n $OOMMEM >> /proc/sys/vm/overcommit_memory
    ;;
    restart|reload|force-reload)
    echo “Error: argument ‘$1’ not supported” >&2
    exit 3
    ;;
    stop)
    # No-op
    ;;
    *)
    echo “Usage: $0 start|stop” >&2
    exit 3
    ;;
    esac
    exit 0

    Information on oom-killer can be found in the Linux Memory Management Wiki.

    To make the confusion most complete, here’s my current settings of these kernel parameters, which may differ from those in the config files. For example I completely disabled oom-killer by setting overcommit_memory to 1.

    scheff@nslu2:~$ cat /proc/sys/vm/swappiness
    0
    scheff@nslu2:~$ cat /proc/sys/vm/laptop_mode
    120
    scheff@nslu2:~$ cat /proc/sys/vm/dirty_writeback_centisecs
    12000
    scheff@nslu2:~$ cat /proc/sys/vm/dirty_expire_centisecs
    12000
    scheff@nslu2:~$ cat /proc/sys/vm/dirty_ratio
    10
    scheff@nslu2:~$ cat /proc/sys/vm/dirty_background_ratio
    1
    scheff@nslu2:~$ cat /proc/sys/vm/overcommit_ratio
    75
    scheff@nslu2:~$ cat /proc/sys/vm/overcommit_memory
    1
    scheff@nslu2:~$ uptime
    21:25:38 up 32 days, 22:56, 1 user, load average: 0.00, 0.00, 0.00

    Well, I hope this will be of some help. Nonetheless stable use of rtorrent (and stupid hpodder is goind oom too) is still a mystery. Good luck!

    Comment by FrankZabbath — 2008/04/06 @ 20:29 | Reply

  3. Thank you for your answer!
    Well, I did some research myself and nothing could solve the OOM mystery…
    It was then that I decided to add the the lenny repo to my /etc/apt/sources.list
    and run:
    apt-get update
    apt-get dist-upgrade

    The procedure, besides all the other packages, updated my kernel to 2.6.24-1-ixp4xx and
    for the past two days the OOM problem has been solved. Rtorrent(0.7.9/libtorrent-0.11.9) is running rock solid, without a single problem and I already downloaded a 7Gb size torrent. During installation I chose to keep my previous conf files. Probably it might have been a kernel issue that was solved.

    Comment by Michael — 2008/04/07 @ 15:34 | Reply

  4. Michael,

    you’re my hero! :) This oom thing was going to drive me nuts. But since finished upgrading to Lenny yesterday, these problems are gone here too! A thousand thanks for letting me know!!

    Upgrade went very smooth using apt-get (aptitude wasn’t happy about dependencies). Only error I got was about Perl Locale defaulting to “C” because of incomplete Locale Variables. And lighttpd refused to start because I obviously had disabled IPv6 in the Kernel Modules and Lighty’s config file had to be changed to not using IPv6.

    Hpodder haven’t ran out of memory so far and my rtorrent downloading a 7 GB torrent is running stable too. Let’s hope upgrading to unstable (Lenny) wouldn’t bring any new severe problems.

    Ciao
    Kristian

    Comment by FrankZabbath — 2008/04/09 @ 19:57 | Reply

  5. Upgraded to Lenny, works perfectly fine for me. Finally rtorrent works!

    Comment by Fritz — 2008/04/21 @ 10:57 | Reply

  6. I think the problem was documented in this thread on LKLM:
    http://linux.derkeiler.com/Mailing-Lists/Kernel/2007-11/msg08223.html

    Apparently, with older kernels, the USB storage system can run out of memory trying to write dirty pages to swap, so the OOM will kick in.

    Comment by Guillaume — 2008/05/06 @ 20:16 | Reply

  7. Hi,

    I know I come here late but I hope not too late.

    I’ve the same problem : oom-killer is killing everything on my NSLU2 (266Mhz) with Debian _Lenny_ !

    I’ve tryied many vm tunings but nothing has worked for me. I’m stucking with that for so long now.

    When oom-killer is triggered I do nothing special but just an rsync from my desktop computer to the slug (big dir with many small files).
    It also triggers when I copy (with cp) a dir with many small files locally on the slug (from slug to slug).

    Some people on the net talk about usb that can cause this type of problem but I don’t know what to do?

    How to diagnose and eventually fix that problem “out of memory” ???

    If you know a better place where I can ask for help let me know … I don’t want to spent your time (and money)

    Sorry for my english I’m french.

    Thank you in advance.

    Comment by Mica — 2009/11/10 @ 20:39 | Reply


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Create a free website or blog at WordPress.com.

%d bloggers like this: