[Xymon] Thoughts on Usefulness/Reliability of Purple Alerts

[J.C. Cleaver]

On Mon, August 24, 2015 10:15 am, Sean MacGuire wrote:
> OK, I'll chime in... I wrote Big Brother so know something about
> purple alerts.
>
> The purple alert was something no other monitoring system did, and
> it took care of the problem of a BB (and Xymon) client dropping dead
> and/or machines being in a zombie state (i.e. responding to pings but
> otherwise hung).
>
> They're useful as a indication of something being wrong, vs a red or
> yellow alert which provide a clear and actionable problem ststus.
>
> So purple alerts are awesome, unless your server has lost contact
> with the clients reporting in and everyone goes purple at the same
> time resulting in the "Massive Purple Explosion".
>
> Background explanation - the idea was to timestamp reports into the
> future, and if a client doesn't report in by then, the validity of
> the last report is in question - it's works the exact same way as
> the expiration date on a carton of milk - the milk might not have
> gone bad, yet, but you might want to check it before drinking it.
>
>
> Matt Vander Werf wrote:
>> This is primarily for Henrik and J.C., but anyone else is free to chime
>> in their thoughts on this as well!
>>
>> *Background:
>> We have a Xymon server (the latest Terabithia RPM on RHEL 7) in
>> production that monitors around 1950 hosts (and consistently growing).
>> About a week ago, we experienced some pretty bad purple alert storms in
>> the middle of the night that were all false-positive alerts (over 300
>> alerts one night). For most of the tests that went purple, they went
>> back to green at the next update interval. At this point, we've been
>> unable to figure out a root cause behind this issue, but it hasn't
>> happened again since early last week (all the easy, understandable
>> possible causes has been ruled out: network load/bandwidth, CPU load of
>> Xymon server and of Xymon clients affected, etc.).
>>
>> We have been using purple alerts for some time now, find them fairly
>> reliable for the most part, and think they are useful, as machines hang
>> or something similar (causing the Xymon client on the machine to stop
>> being able to report to the Xymon server) and we don't get any red or
>> yellow alerts for any other tests (sometimes a machine can hang but
>> still have a network connection that can be successfully pinged by
>> Xymon, we have found). We haven't had any major issues with
>> false-positive purple alerts (for the most part), or any purple alert
>> storms, since we started using them consistently for all our machines a
>> couple years ago.
>>
>> I understand that when Xymon was first forked from Big Brother a long
>> while back, it may have been noted that one big change from Big Brother
>> was that you didn't need to do purple alerts (or something like that)
>> and that it was discouraged to use purple alerts, as they were seen as
>> widely unreliable. (I'm hearing this from a coworker of mine, who set up
>> our original Xymon server some 5 years ago, but have been unable to find
>> what he's referring to.) But from what I can see from the current
>> documentation and the mailing list archives, I'm not seeing any place
>> where the use of purple alerts is discouraged due to them being
>> unreliable.
>>
>> *Question(s):
>> So, I wanted to see what the current thinking/view regarding purple
>> alerts and the use of purple alerts was by both the original main
>> maintainer, Henrik, and the more current main maintainer, J.C. (at least
>> of the current release). Are purple alerts still considered wholly
>> unreliable, or even somewhat unreliable (or were they ever)? Are they
>> discouraged in any way or fashion from being used? Have they caused
>> issues for any of you on this list? Or vice versa: Have they worked well
>> for you? I'm fully aware that this purple alert storm issue we had is
>> just a one-off occurrence and we could have not more additional issues
>> in the future with purple alerts.
>>
>> I understand that purple alerts are different than other alerts, like
>> red and yellow alerts, in that it is an indication that the Xymon client
>> has stopped working/reporting (on a per-test basis) to the Xymon server
>> for some reason, rather than an issue from a specific test (e.g. with
>> the CPU load, memory, etc.).
>>
>> *(Possible) Feature Request:
>> In addition, I'd be interested if there was a way that you could only
>> get one alert for a machine if say all the tests for that machine go
>> purple, instead of an alert for each purple test. I don't believe this
>> is possible currently, correct? Is this something that could possibly be
>> implemented in the future? I understand if it's not or if it wouldn't be
>> very easy.
>>
>>
>> I appreciate your time in answering my questions and look forward to
>> your input! (And apologies for the long-winded e-mail!)
>>
>>
>> Thanks very much in advance!!
>>
>> --
>> Matt Vander Werf


Matt,


Generally speaking, a purple alert should be seen first and foremost as an
indication of a failure in the monitoring *system*... where "system"
includes the client pushing data up from the various servers you're paying
attention to.

By having a calculation made on each message's receipt of how long that
message is good for (receipt time + [default, or specified]), we have a
"fail safe" for an unknown issue occurring that requires attention. The
proximate cause of the purple is the failure to receive a message. Whether
that's caused by a hang or death of the usual originator, a bug in a
xymonproxy, a cut network cable, or xymond being unable to handle all of
the traffic sent to it before it times out, is left somewhat as an
exercise for the administrator.

Because purple alerts are generated from xymond's own view of its internal
state (calculated once a minute) and are never sent IN to xymond, purple
alerts should be a reliable indicator that... some other type of
unreliability is going on :)


Because of the wide possibility of different configurations, it's a little
dangerous to create a one-size-fits-all strategy for purples. In a typical
xymon installation with xymonnet and xymond_client running locally on the
same machine, with no proxies or network segments in the middle, and with
clients reporting directly in as well, you really shouldn't see any purple
alerts outside of clients dying... And if the client is dying because the
box is dying, by default you'll only get the 'conn' test red alert instead
of the various xymond_client and xymonnet-generated ones (unless you're
using the 'noclear' line in hosts.cfg).


Your suggestion to have only a single 'purple' come through would
*typically* work, but you'd have to ask yourself which test would be the
representative one. In our case, we found it easiest to nominate a
specific xymond_client test -- "memory" -- and only send purple
notifications for that out to our alert team. That takes care of
xymond_client, while leaving esoteric situations caused by the failure of
different sharded xymonnet's, xymonproxy's, or custom independent tests
free to fail in their own way.


Again, it's the non-typical cases where it gets tricky. What about custom
tests that aren't being generated by xymond_client that are still
functioning? Perhaps you have xymonnet running on a different machine
that's reporting back to xymond (or to a xymonproxy that's reporting back
to xymond!) that has failed in some way. And of course, it could be that
xymond is under heavy load and is unable to keep up with incoming messages
generally (something we experienced in both the TCP and BFQ configs as we
were scaling out).


Sort of along these lines, however, I'd been considering having a more
"host-wide" way of defining certain failure states directly within xymond,
which would allow some of this override logic to happen centrally (and
more reliably). Imagine a 'conn' being red optionally causing *all* tests
to fail-to-clear, removing the need for this calculation from the
remainder of xymonnet tests. Or a true host-wide "disable" that gets
applied to all tests, even new ones, as a xymond flag. A host-wide
"purple-state" could be conceptualized as well.

That's just a thought, though, and it kind of depends on whether people
would find such a feature useful.


Anyway, I hope that's answered some of your questions!


Regards,

-jc