After switching from Unslung to Debian I constantly got various programs (mt-daapd, rtorrent, gcc) terminating, because they ran out of memory. Tweaking or even disabling the oom-killer kernel feature didn’t do the trick, so I took a look at the ulimits and found most of them set to unlimited. Setting some of the ulimit defaults like on Ubuntu Linux solved these problems.
Update
Thanks to Michael the solution to any oom problem is upgrading to Debian Unstable (aka. Lenny). My “solution” didn’t proof. See the comments for more detail.
After some experimenting, I now have two ulimit settings – stack size and max locked memory – pointed out, but am not sure if really both are mandatory. But it shouldn’t hurt either. These limits, being unlimited by default, now got set to sane values as follows.
You can set the ulimits for the current shell with ulimit -l 32 -s 8192. System wide defaults are defined in /etc/security/limits.conf e.g. by adding this:
#restrictions to avoid oom using ubuntu defaults * hard memlock 32 * hard stack 8192
As mentioned before, I currently have oom-killer disabled. But I should try to enable it again, to benefit from this feature. This can be done with the sysctl tool and in the /etc/sysctl.conf file.
While I’m at it, that would be the output of the stock Debian settings:
$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited max nice (-e) 0 file size (blocks, -f) unlimited pending signals (-i) unlimited max locked memory (kbytes, -l) unlimited max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) unlimited max rt priority (-r) 0 stack size (kbytes, -s) unlimited cpu time (seconds, -t) unlimited max user processes (-u) unlimited virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
That’s the output for Ubuntu Gutsy:
$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 8191 max locked memory (kbytes, -l) 32 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 8191 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
ChangeLog
[2008-04-11: Better upgrade to Lenny.]
Great article! I have the same problem with OOM when I switched from Unslung to Debian. I added the lines to /etc/security/limits.conf:
#restrictions to avoid oom using ubuntu defaults
* hard memlock 32
* hard stack 8192
but this didn’t do the trick for me. Rtorrent is still crashing and dmesg states the OOM issue. Can you be a little more specific on the tweaks I have to make on /etc/sysctl.conf file ?
How much swap does your system use ?
Comment by Michael — 2008/04/04 @ 10:19 |
Hey Michael,
thanks for your reply. Funny enough I had these oom errors again a few days ago. I was trying to download bigger torrents as before, so that seems to be the cause. I can’t limit the resident memory usage via .rtorrent.rc, because that also limits the virtual memory (disk access) and results in downloaded chunks not being written to the disk.
Here’s the current .rtorrent.rc I use on my Slug:
min_peers = 10
max_peers = 80
min_peers_seed = 5
max_peers_seed = 40
max_uploads = 3
upload_rate = 11
directory = /mnt/HDDRIVE2GO/torrents
session = /mnt/HDDRIVE2GO/torrents/.rtorrent_session
schedule = watch_directory,5,5,load_start=/home/scheff/rtorrent-watch/*.torrent
schedule = untied_directory,5,5,stop_untied=
schedule = low_diskspace,5,60,close_low_diskspace=100M
schedule = ratio,60,60,stop_on_ratio=120,125M,250
port_range = 8661-8679
port_random = yes
max_memory_usage = 128M
send_buffer_size = 128K
receive_buffer_size = 1M
max_open_files = 256
max_open_sockets = 512
hash_read_ahead = 5
hash_interval = 10
hash_max_tries = 3
check_hash = no
use_udp_trackers = yes
session_lock = yes
My Slug currently has about 24 MB swapped, that’s without rtorrent running but a couple of other services (mt-daapd, lighttpd with php5, lastfmproxy, kernel-nfs, etc.).
What I was referring to with sysctl.conf are the settings relevant for oom-killer and the so called laptop-mode. On my system these kernel parameters are set by my modified init script kernel-param I found in David Härdeman’s article. It would certainly be more elegant to set these values in sysctl.conf though.
This is my kernel-params files:
#!/bin/sh
PATH=/sbin:/bin
# Max time to wait for writeout
MAX_AGE=120
CENT_AGE=$((100 * $MAX_AGE))
# Max percent of mem to use for dirty pages
DRATIO=10
# Once we write, do so until this many percent of mem is still in use
DBRATIO=1
# Max ram overcommit percentage
OOMRATIO=75
# Overcommit policy
OOMMEM=2
case “$1″ in
start)
echo -n 0 >> /proc/sys/vm/swappiness
echo -n $MAX_AGE >> /proc/sys/vm/laptop_mode
echo -n $CENT_AGE >> /proc/sys/vm/dirty_writeback_centisecs
echo -n $CENT_AGE >> /proc/sys/vm/dirty_expire_centisecs
echo -n $DRATIO >> /proc/sys/vm/dirty_ratio
echo -n $DBRATIO >> /proc/sys/vm/dirty_background_ratio
echo -n $OOMRATIO >> /proc/sys/vm/overcommit_ratio
echo -n $OOMMEM >> /proc/sys/vm/overcommit_memory
;;
restart|reload|force-reload)
echo “Error: argument ‘$1′ not supported” >&2
exit 3
;;
stop)
# No-op
;;
*)
echo “Usage: $0 start|stop” >&2
exit 3
;;
esac
exit 0
Information on oom-killer can be found in the Linux Memory Management Wiki.
To make the confusion most complete, here’s my current settings of these kernel parameters, which may differ from those in the config files. For example I completely disabled oom-killer by setting overcommit_memory to 1.
scheff@nslu2:~$ cat /proc/sys/vm/swappiness
0
scheff@nslu2:~$ cat /proc/sys/vm/laptop_mode
120
scheff@nslu2:~$ cat /proc/sys/vm/dirty_writeback_centisecs
12000
scheff@nslu2:~$ cat /proc/sys/vm/dirty_expire_centisecs
12000
scheff@nslu2:~$ cat /proc/sys/vm/dirty_ratio
10
scheff@nslu2:~$ cat /proc/sys/vm/dirty_background_ratio
1
scheff@nslu2:~$ cat /proc/sys/vm/overcommit_ratio
75
scheff@nslu2:~$ cat /proc/sys/vm/overcommit_memory
1
scheff@nslu2:~$ uptime
21:25:38 up 32 days, 22:56, 1 user, load average: 0.00, 0.00, 0.00
Well, I hope this will be of some help. Nonetheless stable use of rtorrent (and stupid hpodder is goind oom too) is still a mystery. Good luck!
Comment by FrankZabbath — 2008/04/06 @ 20:29 |
Thank you for your answer!
Well, I did some research myself and nothing could solve the OOM mystery…
It was then that I decided to add the the lenny repo to my /etc/apt/sources.list
and run:
apt-get update
apt-get dist-upgrade
The procedure, besides all the other packages, updated my kernel to 2.6.24-1-ixp4xx and
for the past two days the OOM problem has been solved. Rtorrent(0.7.9/libtorrent-0.11.9) is running rock solid, without a single problem and I already downloaded a 7Gb size torrent. During installation I chose to keep my previous conf files. Probably it might have been a kernel issue that was solved.
Comment by Michael — 2008/04/07 @ 15:34 |
Michael,
you’re my hero! :) This oom thing was going to drive me nuts. But since finished upgrading to Lenny yesterday, these problems are gone here too! A thousand thanks for letting me know!!
Upgrade went very smooth using apt-get (aptitude wasn’t happy about dependencies). Only error I got was about Perl Locale defaulting to “C” because of incomplete Locale Variables. And lighttpd refused to start because I obviously had disabled IPv6 in the Kernel Modules and Lighty’s config file had to be changed to not using IPv6.
Hpodder haven’t ran out of memory so far and my rtorrent downloading a 7 GB torrent is running stable too. Let’s hope upgrading to unstable (Lenny) wouldn’t bring any new severe problems.
Ciao
Kristian
Comment by FrankZabbath — 2008/04/09 @ 19:57 |
Upgraded to Lenny, works perfectly fine for me. Finally rtorrent works!
Comment by Fritz — 2008/04/21 @ 10:57 |
I think the problem was documented in this thread on LKLM:
http://linux.derkeiler.com/Mailing-Lists/Kernel/2007-11/msg08223.html
Apparently, with older kernels, the USB storage system can run out of memory trying to write dirty pages to swap, so the OOM will kick in.
Comment by Guillaume — 2008/05/06 @ 20:16 |
Hi,
I know I come here late but I hope not too late.
I’ve the same problem : oom-killer is killing everything on my NSLU2 (266Mhz) with Debian _Lenny_ !
I’ve tryied many vm tunings but nothing has worked for me. I’m stucking with that for so long now.
When oom-killer is triggered I do nothing special but just an rsync from my desktop computer to the slug (big dir with many small files).
It also triggers when I copy (with cp) a dir with many small files locally on the slug (from slug to slug).
Some people on the net talk about usb that can cause this type of problem but I don’t know what to do?
How to diagnose and eventually fix that problem “out of memory” ???
If you know a better place where I can ask for help let me know … I don’t want to spent your time (and money)
Sorry for my english I’m french.
Thank you in advance.
Comment by Mica — 2009/11/10 @ 20:39 |