I've managed to stop the flood of false alerts. I removed all of my non-prod clients from the bb-hosts and shut off their client processes. The problem seems to be somehow related to the amount of data the Xymon server is trying to process. <br>
<br><div class="gmail_quote">On Sun, Jan 17, 2010 at 5:08 PM, Chris Naude <span dir="ltr"><<a href="mailto:chris.naude.0@gmail.com">chris.naude.0@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
I have 7 clients running. Each client has a different name. They are all sending data to the primary Xymon server. The alerts are reading missing processes, full file systems, and msgs errors. Here is another sample of an unusual error. You can see the process list has a funky break in it. <div>
<br></div><div><h3> Sun Jan 17 15:40:18 MST 2010 - Processes NOT ok</h3>
<pre><img src="http://unixadmin.bestwestern.com/xymon/gifs/yellow.gif" alt="yellow" border="0" height="16" width="16"> Expected string COMMAND not found in ps output header
PID PPID USER
STIM] S PRI %CPU TIME VSZ COMMAND
0 0 root Dec 14 S 127 0.16 00:40:00 0 swapper
1 0 root Dec 14 R 152 0.09 00:01:21 2064 init
48 0 root Dec 14 S 152 0.00 00:00:00 0 net_str_cached
45 0 root Dec 14 S 152 0.00 00:00:00 0 net_str_cached
42 0 root Dec 14 S 152 0.00 00:00:00 0 net_str_cached
31 0 root Dec 14 S 152 0.00 00:00:00 0 net_str_cached
30 0 root Dec 14 S 152 0.00 00:00:00 0 net_str_cached
29 0 root Dec 14 S 152 0.00 00:00:00 0 net_str_cached
28 0 root Dec 14 S 152 0.00 00:00:00 0 net_str_cached
26 0 root Dec 14 S 152 0.00 00:00:00 0 net_str_cached
5 0 root Dec 14 R 152 0.00 00:00:02 0 signald
6 0 root Dec 14 R 152 0.00 00:00:03 0 kmemdaemon
17 0 root Dec 14 S 152 0.00 00:00:00 0 net_str_cached
16 0 root Dec 14 S 152 0.00 00:00:00 0 net_str_cached
15 0 root Dec 14 S 152 0.00 00:00:00 0 net_str_cached
14 0 root Dec 14 S 152 0.00 00:00:00 0 net_str_cached
13 0 root Dec 14 S 152 0.00 00:00:00 0 net_str_cached
12 0 root Dec 14 S 152 0.00 00:00:00 0 usbhubd
11 0 root Dec 14 R 152 0.00 00:01:11 0 escsid
10 0 root Dec 14 S -32 0.00 00:00:00 0 ttisr
9 0 root Dec 14 R 152 0.00 00:01:27 0 ksyncer_daemon
7 0]root Dec 14 R 152
0.00 00:]0:00 0 kai_daemon
50 0 root Dec 14 S 152 0.00 00:00:00 0 net_str_cached
47 0 root Dec 14 S 152 0.00 00:00:00 0 net_str_cached
44 0 root Dec 14 S 152 0.00 00:00:00 0 net_str_cached
41 0 root Dec 14 S 152 0.00 00:00:00 0 net_str_cached
</pre><div><div></div><div class="h5"><br><div class="gmail_quote">On Sun, Jan 17, 2010 at 4:21 PM, Josh Luthman <span dir="ltr"><<a href="mailto:josh@imaginenetworksllc.com" target="_blank">josh@imaginenetworksllc.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Is there only one client sending data as this name? I don't think you answered Lars' email.<br><br>What does the alert read and what does the data say? Missing process? Too high of a load?<br><br clear="all">Josh Luthman<br>
Office: 937-552-2340<br>Direct: 937-552-2343<br>1100 Wayne St<br>Suite 1337<br>Troy, OH 45373<br><br>"The secret to creativity is knowing how to hide your sources."<br>--- Albert Einstein<div><div></div><div>
<br>
<br><br><div class="gmail_quote">On Sun, Jan 17, 2010 at 6:11 PM, Chris Naude <span dir="ltr"><<a href="mailto:chris.naude.0@gmail.com" target="_blank">chris.naude.0@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left:1px solid rgb(204, 204, 204);margin:0pt 0pt 0pt 0.8ex;padding-left:1ex">
The problem has suddenly become much much worse. I verified with tcpdump that the data coming from the client is 100% correct. It seems something on the Xymon server side is not handling the client data correctly. Anyone have any other ideas?<div>
<br></div><div><pre><img alt="red" border="0" height="16" width="16"> 89% /testdb3 (37771472% used) has reached the PANIC level (95%)
Filesystem 1024-blocks Used Available Capacity Mounted on
/dev/vgtestdb1/lvol1 107844344 70901816 36942528 66% /testdb1
/dev/vgtestdb2/lvol1 35962064 25453128 10508936 71% /testdb2
/dev/vgtestdb4/lvol1 970909400 825006344 145903056 85% /testdb4
/dev/vgtestdb3/lv
l1 ] 338788224 301016752 37771472 89% /testdb3
/dev/vgtestdb5/lvol1 179789048 150553912 29235136 84% /testdb5
/dev/vg00/lvol8 24580711 74501 24506210 1% /home
/dev/vg00/lvol4 10226680 6339283 3887397 62% /opt
</pre></div><div><div><div></div><div><br><br><div class="gmail_quote">On Sat, Jan 16, 2010 at 10:44 AM, Chris Naude <span dir="ltr"><<a href="mailto:chris.naude.0@gmail.com" target="_blank">chris.naude.0@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left:1px solid rgb(204, 204, 204);margin:0pt 0pt 0pt 0.8ex;padding-left:1ex">
That makes a lot of sense. I did have some issues with the startup scripts on HP-UX. I'll check it out later tonight. Hopefully i can get it fixed before it goes live tonight. Thanks!<div><div></div><div><br>
<br><div class="gmail_quote">On Sat, Jan 16, 2010 at 7:56 AM, Lars Ebeling <span dir="ltr"><<a href="mailto:lars.ebeling@leopg9.no-ip.org" target="_blank">lars.ebeling@leopg9.no-ip.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left:1px solid rgb(204, 204, 204);margin:0pt 0pt 0pt 0.8ex;padding-left:1ex">
<div bgcolor="#ffffff">
<div><font face="Arial" size="2">It looks like two instances of the client are
writing to the file at the same time or almost ;)</font></div>
<div><font face="Arial" size="2"></font> </div><font color="#888888">
<div><font face="Arial" size="2">Lars</font></div></font><div><div></div><div>
<blockquote style="border-left:2px solid rgb(0, 0, 0);padding-left:5px;padding-right:0px;margin-left:5px;margin-right:0px" dir="ltr">
<div style="font-family:arial;font-style:normal;font-variant:normal;font-weight:normal;font-size:10pt;line-height:normal;font-size-adjust:none;font-stretch:normal">----- Original Message ----- </div>
<div style="background:rgb(228, 228, 228) none repeat scroll 0% 0%;font-family:arial;font-style:normal;font-variant:normal;font-weight:normal;font-size:10pt;line-height:normal;font-size-adjust:none;font-stretch:normal">
<b>From:</b>
<a title="chris.naude.0@gmail.com" href="mailto:chris.naude.0@gmail.com" target="_blank">Chris
Naude</a> </div>
<div style="font-family:arial;font-style:normal;font-variant:normal;font-weight:normal;font-size:10pt;line-height:normal;font-size-adjust:none;font-stretch:normal"><b>To:</b> <a title="hobbit@hswn.dk" href="mailto:hobbit@hswn.dk" target="_blank">hobbit@hswn.dk</a> </div>
<div style="font-family:arial;font-style:normal;font-variant:normal;font-weight:normal;font-size:10pt;line-height:normal;font-size-adjust:none;font-stretch:normal"><b>Sent:</b> Saturday, January 16, 2010 4:59
AM</div>
<div style="font-family:arial;font-style:normal;font-variant:normal;font-weight:normal;font-size:10pt;line-height:normal;font-size-adjust:none;font-stretch:normal"><b>Subject:</b> [hobbit] False Process Down
Alerts</div>
<div><br></div>I'm run into a strange problem with my Xymon server. I noticed
today that I'm receiving random false alerts for processes being down. When I
look at the process list output in the alert it looks as if the data coming
from the clients isn't correct. Here is an example. Has anyone seen anything
like this?
<div><br></div>
<div><pre> 9613 1944 root Jan 11 S 154 0.00 00:00:00 6128 cmclconfd -c
10389 1944 root Jan 11 S 154 0.00 00:00:00 6128 cmclconfd -c
9794 1 oracle 10:55:57 S 154 0.00 00:00:0
217600]oracleTEST (LOCAL=NO)
1592 1 oracle Jan 11 S 154 0.00 00:00:11 217136 ora_mman_TEST
12751 1944 root Jan 11 S 154 0.00 00:00:00 6128 cmclconfd -c
8965 1944 root Jan 11 S 154 0.00 00:00:00 6128 cmclconfd -c
</pre>
<div><br></div>
<div><pre>11819 1 oracle Jan 12 S 154 0.00 00:00:07 217280 ora_j015_TEST
2711 1 roo
]ec 4 S 120 0.04 00:02:16 868 /usr/sbin/xntpd
3547 1 xymon Dec 4 S 168 0.00 00:00:43 268 /opt/xymon/client/bin/hobbitlaunch --config=/opt/xymon/client/etc/clientlaunch.cfg --log=/opt/xymon/client/logs/clientlaunch.log --pidfile=/opt/xymon/client/logs/clientlaunch.101.example.com.pid
3728 1 root Dec 4 R 152 0.00 00:00:37 4208 /usr/sbin/stm/uut/bin/tools/monitor/WbemWrapperMonitor
</pre></div>
<div><br></div>
<div><br></div>
<div>Xymon version: 4.3.0-0.beta2</div>
<div>Xymon server: CentOS 5.4 32 bit</div>
<div><br></div>
<div>Client: HP-UX 11.31 Itanium</div>
<div><br>-- <br>Chris Naude<br></div></div></blockquote></div></div></div>
</blockquote></div><br><br clear="all"><br></div></div>-- <br><font color="#888888">Chris Naude<br>
</font></blockquote></div><br><br clear="all"><br></div></div>-- <br>Chris Naude<br>
</div>
</blockquote></div><br>
</div></div></blockquote></div><br><br clear="all"><br></div></div>-- <br>Chris Naude<br>
</div>
</blockquote></div><br><br clear="all"><br>-- <br>Chris Naude<br>