[hobbit] procs test keeps paging, although green for +4 days

[Henrik Stoerner]

On Thu, Aug 17, 2006 at 04:05:38PM +0000, David Gore wrote:
> Interesting problem, we keep getting paged for a procs test on a host 
> twice a day although the procs test has been green for the last 4+ 
> days.  It would appear something is stuck?  What to look at?  It is in 
> couple 'chk' files in ~/server/tmp, but I wouldn't know what to make of 
> that.  Ideas to troubleshoot?

The alert.chk.sub file contains the recipients of the alerts currently
active - there is one line for each recipient. E.g.

   1157011746|myserver|sslcert|mail|henrik at test.com

The alert.chk file has one line for every status that is in a
potentially alerting state, i.e. it is red, yellow or purple. E.g.

myserver|sslcert|mysite/webservers|10.0.36.166|yellow|1155640943|1157011746|paging|status ...

The field that says "paging" has "norecip" if the status doesn't have
any alert-recipients defined, or if the alerts are restricted, e.g. to
a certain time of day.


I've never seen it happen, but there is a very small time window between
the startup of the hobbitd daemon and the startup of the hobbitd_alert
module where a green update is registered with hobbitd, but it doesn't
make it to the hobbitd_alert module - and then you have this situation.

Restarting the hobbitd_alert module - just kill the hobbitd_alert
process, it will restart automatically - should clean it up, logging a
message like "Stale alert for HOSTNAME:TEST dropped" to the page.log
file.


Regards,
Henrik