Alerting on "non-green" events of a given duration

[Walter Appleby]

Scenario:
       We have several mail servers that are doing spam filtering.  When they receive a spam definition update there is a brief recycle of the smtp services.  This causes Hobbit's smtp test to go "colorful" (red or yellow).  
       To avoid being notified every time there was a momentary blip due to this update cycle I setup an alert rule as follows:
            PAGE=mail_servers
                     MAIL <my address at my domain> SERVICE=smtp DURATION>2m
   
  This works as desired by suppressing alerts for times when the smtp service goes from green to red or yellow and then back to green in less than two minutes.
   
   
  PROBLEM:
       The above rule does not alert me when the smtp service goes from green to red for 1 min, then red to yellow for 1 min, then yellow back to red for 1 min... etc.  In this case since no one event is longer than two minutes I'm not getting alerts even though my servers are non-green for a longer period of time.  The way I understand the DURATION keyword is that it keeps a timer on the present state (red, yellow, purple) and that when the state changes the timer is reset.  What I want is a way to keep track of the total duration of non-green events.  Whereas a blip when the smtp service recycles is acceptable,  I need to know if it struggles to return to green afterward.
   
  DESIRED STATE:
       I want to be alerted when the smtp service is anything other than green for more than two minutes.  This could be a single red or yellow event lasting more than two minutes or it could be a string of connected red and yellow events that when combined are greater than two minutes in length.
   
  Any help you can offer is appreciated.  If you need more info or clarification please let me know.
   
  Thanks,
  Danny

       
---------------------------------
Be a better friend, newshound, and know-it-all with Yahoo! Mobile.  Try it now.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xymon.com/pipermail/xymon/attachments/20080219/c02e2cb0/attachment.html>