[hobbit] 'red' status not showing up in history

[Menton, Stephen]

Well, the server was restarted back on the 4th... Current procs:
  hobbit 12332 12230  0   Mar 04 ?        5:26 hobbitd_channel
--channel=page --log=/var/log/hobbit/page.log hobbitd_alert --c
  hobbit 12231 12230  2   Mar 04 ?       379:51 hobbitd
--pidfile=/var/log/hobbit/hobbitd.pid --restart=/opt/home/hobbit/server
  hobbit 12326 12319  0   Mar 04 ?       40:04 hobbitd_rrd
--rrddir=/opt/home/hobbit/data/rrd
  hobbit 12328 12321  0   Mar 04 ?        6:02 hobbitd_client
  hobbit 12333 12330  2   Mar 04 ?       117:51 hobbitd_history
  hobbit 12321 12230  0   Mar 04 ?        1:44 hobbitd_channel
--channel=client --log=/var/log/hobbit/clientdata.log hobbitd_c
  hobbit 12331 12230  0   Mar 04 ?        0:15 hobbitd_channel
--channel=clichg --log=/var/log/hobbit/hostdata.log hobbitd_hos
  hobbit 12334 12331  0   Mar 04 ?        0:25 hobbitd_hostdata
  hobbit 12327 12320  0   Mar 04 ?        4:43 hobbitd_rrd
--rrddir=/opt/home/hobbit/data/rrd
  hobbit 12330 12230  0   Mar 04 ?        0:04 hobbitd_channel
--channel=stachg --log=/var/log/hobbit/history.log hobbitd_hist
  hobbit 12320 12230  0   Mar 04 ?        0:12 hobbitd_channel
--channel=data --log=/var/log/hobbit/rrd-data.log hobbitd_rrd -
  hobbit 12319 12230  0   Mar 04 ?       21:28 hobbitd_channel
--channel=status --log=/var/log/hobbit/rrd-status.log hobbitd_r
  hobbit 12335 12332  1   Mar 04 ?       153:22 hobbitd_alert
--checkpoint-file=/opt/home/hobbit/server/tmp/alert.chk --checkpo

hobbitd_history seems to be running...

Again, status of other tests are being reflected properly, history
recorded properly, both for the same test on other hosts AND for
different tests on the same host. It's like random histories are being
ignored... x,x

history.log shows some errors... But they are later than confirmed times
when some histories weren't being recorded (~2 days ago).

hobbitlaunch.log shows a termination of hobbitd 2 days ago... But this
was the time of a restart so perhaps it's expected. Again, it has been
writing valid histories for some items since then. I'll try forcibly
shutting down the server and brining it back up as well.

stephen


-----Original Message-----
From: Henrik Stoerner [mailto:henrik at hswn.dk] 
Sent: Thursday, March 06, 2008 1:43 PM
To: hobbit at hswn.dk
Subject: Re: [hobbit] 'red' status not showing up in history

On Thu, Mar 06, 2008 at 10:56:16AM -0800, Menton, Stephen wrote:
> I have a test that's gone 'red' multiple times today. I know this 
> because I've seen it in the website and alerts have been sent to me. 
> Yet if I look in ~hobbit/data/hist, ~hobbit/data/histlogs, or use the 
> hsitory CGIs in the web GUI, it shows that it hasn't been 'red' today,

> rather that it's been green for >2 days.

The only explanation I can give is that the hobbitd_history module might
have been stopped or crashed when this red status happened.
Could you check the history.log and hobbitlaunch.log files to see if
there's any mention of this ?

Doing a full restart of Hobbit will force a sync of the history logs
with the current status recorded in Hobbit, but it obviously cannot
record events that are long past.


Regards,
Henrik


To unsubscribe from the hobbit list, send an e-mail to
hobbit-unsubscribe at hswn.dk