[Xymon] Phantom red statuses (Fwd: Xymon [750466] mgmtconsole:msgs CRITICAL (RED))

[J.C. Cleaver]

Hi Greg,

The flapping warning is what tips it off. Flap-detection in xymond
functions by looking at alternating alert states (eg, red/green) happening
within a certain period and "pegging" it at the higher status while it's
going back and forth. This prevents spurious recovery messages and
untimely pager death.

The thing is, though, for a normal functioning 'msgs' test it's almost
impossible for it to actually flap out of the box. The logfetch program
which controls the raw data sent to xymond_client for evaluation actually
walks back 6 "periods" (run cycles) in the log file and sends all
subsequent data up to xymond. This helps with mitigating any lost messages
by turning the 'msgs' test into a "Recent Errors in the Log" test instead
of a direct reflection of the event, since xymon is a state-based
monitoring system rather than a single-fire-and-forget (trap) based
system.

Out of the box, a single red event will cause the msgs test to remain red
for a solid 30m -- far too long for flapping to get triggered in most
cases.


Is there any chance you have multiple servers reporting in with the name
'mgmtconsole'? Especially if you're not using FQDN (which it doesn't seem
like you are), that seems like something that might cause this: Two
different servers with the same name, each sending their own red/green
states every few minutes.


HTH,
-jc



On Thu, February 11, 2016 2:15 pm, Greg Earle wrote:
> I'm running Xymon 4.3.12-2 server (yeah, I know ...) on my management
> system.
> (RHEL 6.5 currently)
>
> A couple of days ago I migrated our central syslog server over to the
> Xymon server, so now "/var/log/messages" is getting a ton of stuff in
> it that it never had before since all my systems are now reporting in
> to it.
>
> Ever since then I've seen something weird - every hour (for about 17+
> hours)
> I was getting RED alerts for the management console's own "msgs" status,
> but
> the actual e-mail notifications don't show anything marked red in them!
>
> It's either yellow or, as in the forwarded message below, green.  I have
> no
> idea why I was getting RED alerts for this file if it thinks it's yellow
> or
> green - any ideas?
>
> The only other thing I can add is that when I go to the Web page for
> mgmtconsole:msgs, it says "WARNING: Flapping status" at the top.
>
> Is that a clue?
>
> (Update: interestingly, it looks like the status has finally changed to
>  green a few minutes ago - after having been red for nearly 17 1/2 hours.
>  Still seeing "WARNING: Flapping status" on the svcstatus Web page,
> though.)
>
> Thanks,
>
> 	- Greg
>
>> Begin forwarded message:
>>
>> From: xymon Monitor <xymon at mgmtconsole.my.do.ma.in>
>> Subject: Xymon [750466] mgmtconsole:msgs CRITICAL (RED)
>> Date: February 11, 2016 at 11:57:25 AM PST
>> To: sysadmins at mgmtconsole.my.do.ma.in
>>
>> green Thu Feb 11 11:57:24 PST 2016 - Log files ok
>> <pre>
>> </pre>
>>
>> No entries in <a
>> href="/xymon-cgi/svcstatus.sh?CLIENT=mgmtconsole&SECTION=msgs:/var/log/messages">/var/log/messages</a>
>>
>>
>> Full log <a
>> href="/xymon-cgi/svcstatus.sh?CLIENT=mgmtconsole&SECTION=msgs:/var/log/messages">/var/log/messages</a>
>> <...SKIPPED...>
>> Feb 11 11:57:17 host7 nrpe[20194]: [ID 927837 daemon.info] connect from
>> mtfuji
>>
>> [... rest elided ... ]
>>
>> See
>> http://mgmtconsole/xymon-cgi/svcstatus.sh?HOST=mgmtconsole&SERVICE=msgs
>
> _______________________________________________
> Xymon mailing list
> Xymon at xymon.com
> http://lists.xymon.com/mailman/listinfo/xymon
>