[Lugstuff] shared memory kernel

[Thomas Gallen]

Paging hell refers to (I believe) the kernel swapping out pages and
swapping in pages (page in/page out) rapidly due to programs allocating
too much memory (again, I believe). The translation between virtual
memory addresses and physical memory addresses allows Linux to give
programs the illusion that there's much more memory available than
there actually is. The intent is to allow the kernel to manage the
allocation, deallocation and swapping of chunks of continuous pages
transparently so that the user doesn't need to know that the seemingly
linear memory chunk they have allocated may actually be spread out
between multiple pages that are not necessarily linear in physical
memory. The additional more obviously consequence of swapping pages
into and out of swap is that, as it's bound to a physical hard disk,
the performance impact is rather horrific.

On another note, unless I'm mistaken as to the meaning of the
statistics, vmstat will allow you to monitor the page in and page out
(though I think it calls them swap in (si) and swap out (so)) rates as
currently reported by the kernel so if you want to try playing with the
sysctl values while doing some load tests and keeping a close eye on
vmstat then perhaps that may help you find the sweet spot you're
looking for.

Disclaimer: I'm not a Linux kernel developer am still trying to wrap my
head around all of the Linux memory management stuff myself. Some
interesting reading:

http://scriptmatrix.net/cs2106/wiki/index.php?title=Buddy_Allocator
http://linux-mm.org/PageAllocation

Thomas

On Thu, Sep 16, 2010 at 05:58:57PM -0400, Theodore Knab wrote:
> Thanks Thomas,
> 
> I saw that article before, but I re-read it again.
> 
> It seems paging hell seems to be the result of setting too much shared
> memory.
> 
> But, what is paging hell ?
> 
> 
> On 16/09/10 09:31 -0400, Thomas Gallen wrote:
> > I believe this will answer most of your questions:
> > 
> > http://www.pythian.com/news/245/the-mysterious-world-of-shmmax-and-shmall/
> > 
> > Thomas
> > 
> > On Sep 16, 2010, at 8:43 AM, Theodore Knab wrote:
> > 
> > > Hi I was wondering if anyone knows how to tune the three shared memory parameters
> > > found in the /etc/sysctl.conf.  For those of you not familiar with
> > > sysctl, it holds tunable kernel configurations. 
> > > 
> > > http://en.wikipedia.org/wiki/Sysctl
> > > 
> > > Normally, you have to increase the default settings for shared
> > > memory use when messing with database applications and other applications that lock a lot
> > > of memory.
> > > 
> > > The three I am focusing on are:
> > > kernel.shmax
> > > kernel.shmall
> > > kernel.shmni
> > > 
> > > More specifically, I am wondering what happens if the tuning numbers for
> > > shmax are set too high ? Does the kernel think it has more RAM that it
> > > does ? The reverse seems to be true. For example, if the shared memory
> > > number is too low applications requiring more shared memory will crash
> > > or just not start.
> > > 
> > > For example, my home system runs Zone Minder for keeping track of
> > > camera data. The default settings for shmax of 32Mb had to be 
> > > increased for this application to work.
> > > 
> > > In contrast, I found a Informix machine that was tuned with a
> > > shmmax that I thing was outrageously high.
> > > 
> > > For example, I found a system with 24GB had this parameter:
> > > kernel.shmmax = 4398046511104 
> > > 
> > > According to the Google calculation this number translates to:
> > > 4 398 046 511 104 bytes = 4096 gigabytes
> > > 
> > > This is the default setting that IBM Informix has on their website for
> > > their Database application. So, I can understand how this may have been 
> > > set. However, it seems not many people would have 4096GB or more on their machine.
> > > 
> > > http://publib.boulder.ibm.com/infocenter/idshelp/v115/index.jsp?topic=/com.ibm.relnotes.doc/ids_1150xc5/mach/ids_machine_notes_11.50.linppc64.html
> > > 
> > > I have since reset the setting to 75% of the total RAM in bytes, but I am still
> > > wondering what happens to a Linux system. The machine in question locked
> > > up twice due to all the memory being used. This is not something that
> > > created a log entry. Also, the machine seemed to become unstable when it started swapping. 
> > > It was just as if all the applications on the machine were trying to consuming more RAM 
> > > that the system had.  For example, recently while running a database upgrade script on the
> > > machine, I could not run simple shell commands. When I tried to log in,
> > > I could type a few characters but only after a few minutes of waiting.
> > > I tried to type 'ctrl-alt-del' but it appeared the program the shutdown
> > > program could not be run in the time I had available to wait.  
> > > It was not a hard crash where errors appear. 
> > > 
> > > Could this be a possible symptom of the kernel.shmmax being too high ?
> > > 
> > > Here are the current setting I am using for the 24GB machine.
> > > kernel.shmmax = 18956485632 
> > > kernel.shmall = 4194304 
> > > kernel.shmmni = 4096
> > > 
> > > I used this script to calculate the max shared memory:
> > > #!/bin/sh
> > > #limits the kernel's shared memory to 1/2 actually physical memory
> > > mem_bytes=`awk '/MemTotal:/ { printf "%0.f",$2 * 1024}' /proc/meminfo`
> > > mem_max=`expr $mem_bytes / 2` #devide by 2
> > > mem_max=`expr $mem_bytes / 4` #devide by 3
> > > 
> > > echo "mem max is $mem_max"
> > > 
> > > mem_max=`expr $mem_max + $mem_max + $mem_max` #gets a 75%
> > > page_size=`getconf PAGE_SIZE`
> > > shmall=`expr $mem_bytes / $page_size`
> > > echo \# Maximum shared segment size in bytes
> > > echo kernel.shmmax = $mem_max
> > > echo \# Maximum number of shared memory segments in pages
> > > echo kernel.shmall = $shmall
> > > 
> > > 
> > > -- 
> > > Ted Knab
> > > 
> > > _______________________________________________
> > > Lugstuff mailing list
> > > Lugstuff at annapolislinux.org
> > > http://list.annapolislinux.org/cgi-bin/mailman/listinfo/lugstuff
> > 
> 
> -- 
> Ted Knab
> Stevensville, MD USA
> 
> _______________________________________________
> Lugstuff mailing list
> Lugstuff at annapolislinux.org
> http://list.annapolislinux.org/cgi-bin/mailman/listinfo/lugstuff