<div dir="ltr"><div>Ah, okay. That makes sense! I guess I was thinking that this only happened with other network tests and not the normal tests as well, but that appears to be a different setting. I must have misread or missed the noclear option when I was looking through the documentation....<br></div><div><br></div>I think maybe this should be documented better...something like if you use noping then you should also use noclear if you want purple statuses. But that's just my thinking...<br><div class="gmail_extra"><br></div><div class="gmail_extra">Thanks as always, J.C.!!<br><br clear="all"></div><div class="gmail_extra"><div><div class="gmail_signature"><div>--</div><div>Matt Vander Werf</div></div></div>
<br><div class="gmail_quote">On Thu, Dec 3, 2015 at 12:49 PM, Japheth Cleaver <span dir="ltr"><<a href="mailto:cleaver@terabithia.org" target="_blank">cleaver@terabithia.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"><div><div class="h5">
<div>On 12/3/2015 9:43 AM, Matt Vander Werf
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Hello,<br>
<br>
</div>
<div>I am having an issue with Xymon where instead of tests
going purple when the client stops reporting, the tests are
going clear.<br>
<br>
</div>
<div>I noticed this with a host that had all it's tests go clear
instead of purple. Turns out the network interface on the
machine had completely died and this had happened a week ago!
We never noticed because instead of going purple the tests for
the machine went clear!<br>
<br>
</div>
<div>This seems to only be an issue with a certain group of
machines. For this group of machines, we have the ping test
disabled by using the 'noping' option on all of them. This is
because they are all behind a firewall with private IP
addresses so they are unable to be contacted by the Xymon
server. But they can still send client data out to the Xymon
server.<br>
<br>
</div>
<div>Turns out, ever since we started using the 'noping' option
for all of them, none of the machines have ever gone purple...<br>
<br>
</div>
<div>I tested this by stopping the xymon-client service on one
of the machines in question, and sure enough, after the
STATUSLIFETIME time limit, all the tests for that host went
clear, instead of going purple.<br>
<br>
<br>
</div>
<div>I looked through the different logs (I already had most set
in debug mode for a different reason), and I didn't see much
that would explain this (but I could have missed something).<br>
<br>
</div>
<div>I did notice in the xymond log file that, according to
xymond, they should have been going purple and not clear.<br>
<br>
</div>
<div>Here's an excerpt from that log file (this is the machine
which I stopped the service on):<br>
<br>
9680 2015-12-02 09:57:48.040111 -> check_purple_status<br>
9680 2015-12-02 09:57:48.047630 Purple log from <HOST>
memory<br>
9680 2015-12-02 09:57:48.047674 ->handle_status<br>
9680 2015-12-02 09:57:48.047676 modifyonly = 0, changed = 0<br>
9680 2015-12-02 09:57:48.047680 - sum: 0, synced: 0,
oldcolor: 0, newcolor: 1, modifychanged: 0<br>
9680 2015-12-02 09:57:48.047682 posting to stachg channel:
host=<HOST>, test=memory<br>
9680 2015-12-02 09:57:48.047684 -> posttochannel<br>
9680 2015-12-02 09:57:48.047697 Posting message 14359 to 1
readers<br>
9680 2015-12-02 09:57:48.047703 <- posttochannel<br>
9680 2015-12-02 09:57:48.047705 posting to status channel<br>
9680 2015-12-02 09:57:48.047706 -> posttochannel<br>
9680 2015-12-02 09:57:48.047712 Posting message 72429 to 2
readers<br>
9680 2015-12-02 09:57:48.047726 <- posttochannel<br>
9680 2015-12-02 09:57:48.047727 <-handle_status<br>
<br>
</div>
<div>Basically this showed up for all the different tests for
this machine.<br>
</div>
<div><br>
</div>
<div>And here's the event log for the same machine:<br>
<br>
Wed Dec 2 09:57:48 2015 <HOST>
cpu
<img src="https://mon.crc.nd.edu/xymon/gifs/green.gif" alt="green" title="green" height="16" border="0" width="16">
<img src="https://mon.crc.nd.edu/xymon/gifs/arrow.gif" alt="From
-> To" border="0">
<a href="https://mon.crc.nd.edu/xymon-cgi/historylog.sh?HOST=jim.vectorbase.org&SERVICE=cpu&TIMEBUF=1449068268" target="_blank"></a><img src="https://mon.crc.nd.edu/xymon/gifs/clear.gif" alt="clear" title="clear" height="16" border="0" width="16"><span style="background-color:rgb(255,255,255)"></span></div>
<div>
<div>
<div>
<div>
<div>
<div dir="ltr">
<div>Wed Dec 2 09:57:48 2015 <HOST> disk <img src="https://mon.crc.nd.edu/xymon/gifs/green.gif" alt="green" title="green" height="16" border="0" width="16">
<img src="https://mon.crc.nd.edu/xymon/gifs/arrow.gif" alt="From -> To" border="0">
<a href="https://mon.crc.nd.edu/xymon-cgi/historylog.sh?HOST=jim.vectorbase.org&SERVICE=disk&TIMEBUF=1449068268" target="_blank"></a><img src="https://mon.crc.nd.edu/xymon/gifs/clear.gif" alt="clear" title="clear" height="16" border="0" width="16"><br>
Wed Dec 2 09:57:48 2015 <HOST>
inode
<img src="https://mon.crc.nd.edu/xymon/gifs/green.gif" alt="green" title="green" height="16" border="0" width="16">
<img src="https://mon.crc.nd.edu/xymon/gifs/arrow.gif" alt="From -> To" border="0">
<a href="https://mon.crc.nd.edu/xymon-cgi/historylog.sh?HOST=jim.vectorbase.org&SERVICE=inode&TIMEBUF=1449068268" target="_blank"></a><img src="https://mon.crc.nd.edu/xymon/gifs/clear.gif" alt="clear" title="clear" height="16" border="0" width="16"><br>
Wed Dec 2 09:57:48 2015 <HOST>
memory
<img src="https://mon.crc.nd.edu/xymon/gifs/green.gif" alt="green" title="green" height="16" border="0" width="16">
<img src="https://mon.crc.nd.edu/xymon/gifs/arrow.gif" alt="From -> To" border="0">
<img style="margin-right:0px" src="https://mon.crc.nd.edu/xymon/gifs/clear.gif" alt="clear" title="clear" height="16" border="0" width="16">
<br>
<br>
<br>
</div>
<div>Any thoughts as to what's going on? Looks like
a bug to me...<br>
<br>
</div>
<div>Thanks!!<br>
<div>
<div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br></div></div>
This is intentional. It's a result of the normal 'purple' behavior
when a box is actually down, but it'll give you this behavior if
you're testing a box that's "normally" conn-down (as far as xymon
knows) anyway.<br>
<br>
You'll want to add the 'noclear' line as well to any of the systems
that are not pingable if you want the client tests to actually go
purple.<br>
<br>
<a href="https://www.xymon.com/help/manpages/man5/hosts.cfg.5.html#lbAG" target="_blank">https://www.xymon.com/help/manpages/man5/hosts.cfg.5.html#lbAG</a><i><br>
</i>
<blockquote><i>noclear</i><br>
<i>Controls whether stale status messages go purple or clear when
a host is down. Normally, when a host is down the client
statuses ("cpu", "disk", "memory" etc) will stop updating - this
would usually make them go "purple" which can trigger alerts. To
avoid that, Xymon checks if the "conn" test has failed, and if
that is true then the other tests will go "clear" instead of
purple so you only get alerts for the "conn" test. If you do
want the stale statuses to go purple, you can use the "noclear"
tag to override this behaviour.</i></blockquote>
<br>
Regards,<br>
-jc<br>
<br>
<br>
</div>
</blockquote></div><br></div></div>