notification for red alert missing

[Dominique Frise <Dominique.Frise (at) unil.ch>]

Hi,

Following red alarm did not generate any notification (no mail/SMS).
There are no DOWNTIME set; the host was not disabled (transitions green -> red 
and red -> green show up in eventlog)


[bb (at) iris conn]$ pwd
/soft/pub/BB/hobbit/data/histlogs/enzo/conn

[bb (at) iris conn]$ cat Thu_Jan_11_11\:54\:15_2007
red <!-- [flags:ordAstLe] --> Thu Jan 11 11:53:59 2007 conn NOT ok

Service conn on enzo2 is not OK : Host does not respond to ping


System unreachable for 1 poll periods (0 seconds)

&red 1.2.3.4 is unreachable
Traceroute results:
traceroute: Warning: Multiple interfaces found; using 1.2.3.5 @ ce0
traceroute to 1.2.3.4 (1.2.3.4), 30 hops max, 40 byte packets
 1  enzo2.unil.ch (1.2.3.4)  1.419 ms !X  0.245 ms !X  0.228 ms !X




Status unchanged in 0.00 minutes
Message received from 130.223.27.23
Client data ID 1168512627
----------------------

Here the red/conn rule test for this host:

[bb (at) iris bin]$ ./hobbitd_alert --test enzo2 conn --color=red | egrep 
"\*\*\*|mail|sms"
00005043 2007-01-11 13:06:22 *** Match with 
'HOST=argos,bigmac,docusurf,eliot,enzo,enzo2,fmp7web,ged,hippocrate,iris,mailc,mailc1,mailc2,midas,zeb1,zeb2,zelda,zelda2 
EXHOST=wit-system' ***
00005043 2007-01-11 13:06:22 *** Match with '$MAIL_DF' ***
00005043 2007-01-11 13:06:22 Mail alert with command 'mail -s "Hobbit [12345] 
enzo2:conn CRITICAL (RED)" x.y (at) z.ch'
00005043 2007-01-11 13:06:22 *** Match with '$SMS_DF' ***
00005043 2007-01-11 13:06:22 Script alert with command 
'/soft/pub/BB/hobbit/server/bin/sendsms' and recipient 1234567890
00005043 2007-01-11 13:06:22 *** Match with 'PAGE=%^$ 
EXHOST=$ADMIN_HOSTS,esope,pcsan COLOR=red' ***
00005043 2007-01-11 13:06:22 *** Match with '$MAIL_DH' ***
00005043 2007-01-11 13:06:22 Mail alert with command 'mail -s "Hobbit [12345] 
enzo2:conn CRITICAL (RED)" a.b (at) c.ch'
00005043 2007-01-11 13:06:22 *** Match with 'HOST=*' ***
00005043 2007-01-11 13:06:22 *** Match with '$MAIL_ROOT UNMATCHED' ***



There are no trace of the event in notification.log :-(

Where to start looking?

Dominique
UNIL - University of Lausanne