[Xymon] Solaris 10 swap not showing on memory alert correctly?

[Gore, David W (David)]

Just a heads up.  So unfortunately, I am going to have to try something different.  One of my xymon scripts got hung and launched every 5 minutes and filled /tmp and swap was consumed as reported by swap -s but not swap -l.  Production processes could not allocate memory and Xymon was blissfully unaware reporting 0 swap used and memory Ok.  

Here is a message from the log (/var/adm/messages). Of course, we added this message to our alert list.

Aug 24 15:10:24 ourHost genunix: [ID 470503 kern.warning] WARNING: Sorry, no swap space to grow stack for pid 19464 (ksh)

swap as shown by Xymon at 0 used
memory less than 50% of 128G used

Xymon client 4.3.10
Xymon server 4.3.12

Server monitored is a SUN T5240 running Solaris 10

Ok, I just realized this is a bit odd isn't it? Out of swap but plenty of physical memory left.  Time to Google.

David W Gore

From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Gore, David W (David)
Sent: Thursday, June 27, 2013 9:06 AM
To: Vernon Everett
Cc: xymon at xymon.com
Subject: Re: [Xymon] Solaris 10 swap not showing on memory alert correctly?

That link appears  to only apply to Zones as I can see the numbers roughly adding up similarly on at least one of my Solaris machines allocated as a zone.  Here is how I think the numbers add up on my non-zone traditional machine.

prtconf|grep Memory
Memory size: 24576 Megabytes

Physical memory: 25,769,803,776 bytes

swap -s
total: 18615728k bytes allocated + 3112800k reserved = 21728528k used, 32078664k available

Converted to bytes:
(x1000)total: 18,615,728,000 bytes allocated + 3,112,800,000 reserved = 21,728,528,000 used, 32,078,664,000 available
(x1024)           19,062,505,472 +                                 3,187,507,200 =                     22,250,012,672 used, 32,848,551,936 available

swap -l
swapfile             dev  swaplo blocks   free
/dev/vx/dsk/bootdg/swapvol 278,60000     16 73552112 72369616

Let's look at the last two numbers 73,552,112*512 = 37658681344 bytes    72,369,616*512= 37,053,243,392  37,658,681,344 -37,053,243,392 = 605,437,952 == 577M swap used

So to answer my own question.  I think the first 2 numbers in swap -s represent physical memory.  I think the last number is some amount from swap and some amount from physical memory.  Some of what's in Vernon's article hints at where it all gets allocated.

It would appear that Xymon just subtracts the last two numbers in swap -l to get swap used in megabytes.


~David

From: Vernon Everett [mailto:everett.vernon at gmail.com] 
Sent: Wednesday, June 26, 2013 11:00 PM
To: Gore, David W (David)
Cc: xymon at xymon.com
Subject: Re: [Xymon] Solaris 10 swap not showing on memory alert correctly?

Solaris and swap is tricky to nail down, specifically because different commands will report different values, by calculating the values in different ways.
You need to choose your commands carefully, depending on what value you need to know.

This might help explain things.
http://www.f3partners.com/blog/bid/49584/Solaris-Swap-Q-A
It all becomes rather fuzzy when you consider what ZFS is doing in your memory.
Also, in a zone, all bets are off.
Caps in particular, on memory or swap, can completely muddy the water.
All the checks within the zone, will show the max available as being the capped value.
However when you interrogate the kernel for amount free, it returns what the kernel can see. And the kernel can see everything, because it exists in the global zone.

Regards
Vernon

P.S. WARNING: Thinking too hard about this can cause your brain to melt. :)


On 27 June 2013 00:38, Gore, David W (David) <david.gore at verizon.com> wrote:
Wed Jun 26 16:26:48 GMT 2013 - Memory low
   Memory              Used       Total  Percentage
yellow Physical          23355M      24576M         95%
green Swap                106M      35913M          0%
 
>From the client data:
 
[swap]
total: 18459352k bytes allocated + 3036456k reserved = 21495808k used, 32308920k available
 
I would expect to see the 21495808k used number?
 
Where is it getting the 106M number?  Anyone else seen this on Solaris?  
 
~David
 

_______________________________________________
Xymon mailing list
Xymon at xymon.com
http://lists.xymon.com/mailman/listinfo/xymon



-- 
"Accept the challenges so that you can feel the exhilaration of victory"
- General George Patton