bbgen frequent yellow alerts - hobbitd problem?

[Mr-Pope]

Hi,

We are running a new installation of Hobbit 4.2 on Solaris 10 running
in a non-global zone.  Server is a v240 but I don't think that matters
here.

The problem here is that our bbgen status turns yellow with fairly
high frequency, sometimes multiple times an hour, at (what seem like)
random intervals.  In the yellow alert bbgen reports:
"hobbitd status-board not available"

During this time the hobbitd daemon is still running and the next time
that bbgen runs the alert (usually) turns green.  I've tested this by
running bbgen every second, every 15 seconds, and every minute.  The
same is also true if I run bbgen by hand.

During the 'yellow alert' time window the bb2.html gets updated with
"All Monitored Systems OK"
When all monitored systems are NOT OK.  When the status turns green
again this page reflects the correct status for the non-green systems.

Below are the output from some commands/logs.  These logs don't really
seem to help, so let me know if there is anything else that I can send
along to debug this issue.

Any help is appreciated - we're near the point of frustration to where
we may have to pull the plug on Hobbit and go back to our old BB
installation.

Thanks in advance.
-Jon

(logs below)


hobbitd log from --debug.  Way less entries here than normal.
2006-11-03 10:54:00 -> do_message/1 (12 bytes): hobbitdboard
2006-11-03 10:54:00 -> update_statistics
2006-11-03 10:54:00 <- update_statistics
2006-11-03 10:54:00 -> oksender
2006-11-03 10:54:00 <- oksender(1-a)
2006-11-03 10:54:00 -> setup_filter: hobbitdboard
2006-11-03 10:54:00 <- setup_filter: hobbitdboard
2006-11-03 10:54:00 <- do_message/1
2006-11-03 10:54:01 -> do_message/1 (0 bytes):
2006-11-03 10:54:01 -> update_statistics
2006-11-03 10:54:01 <- update_statistics
2006-11-03 10:54:01 <- do_message/1



$BB --debug $BBDISP "hobbitdboard"
(with no --debug on a 'failure' I get no output.  I'm assuming this is
the same cause of the bbgen yellow alert)

2006-11-03 10:54:01 Transport setup is:
2006-11-03 10:54:01 bbdportnumber = 1984
2006-11-03 10:54:01 bbdispproxyhost = NONE
2006-11-03 10:54:01 bbdispproxyport = 0
2006-11-03 10:54:01 Recipient listed as '10.xxx.xxx.xxx'
2006-11-03 10:54:01 Standard BB protocol on port 1984
2006-11-03 10:54:01 Will connect to address 10.xxx.xxx.xxx port 1984
2006-11-03 10:54:01 Connect status is 0
2006-11-03 10:54:01 Sent 12 bytes
2006-11-03 10:54:01 Closing connection


bbgen --debug --report (this one turned bbgen yellow/unavailable.
Note the quick disconnect.)
2006-11-03 09:51:03 load_state()
2006-11-03 09:51:03 Transport setup is:
2006-11-03 09:51:03 bbdportnumber = 1984
2006-11-03 09:51:03 bbdispproxyhost = NONE
2006-11-03 09:51:03 bbdispproxyport = 0
2006-11-03 09:51:03 Recipient listed as '10.xxx.xxx.xxx'
2006-11-03 09:51:03 Standard BB protocol on port 1984
2006-11-03 09:51:03 Will connect to address 10.xxx.xxx.xxx port 1984
2006-11-03 09:51:03 Connect status is 0
2006-11-03 09:51:03 Sent 126 bytes
2006-11-03 09:51:03 Closing connection

bbgen --debug --report (this one worked fine)
2006-11-03 09:54:00 load_state()
2006-11-03 09:54:00 Transport setup is:
2006-11-03 09:54:00 bbdportnumber = 1984
2006-11-03 09:54:00 bbdispproxyhost = NONE
2006-11-03 09:54:00 bbdispproxyport = 0
2006-11-03 09:54:00 Recipient listed as '10.xxx.xxx.xxx'
2006-11-03 09:54:00 Standard BB protocol on port 1984
2006-11-03 09:54:00 Will connect to address 10.xxx.xxx.xxx port 1984
2006-11-03 09:54:00 Connect status is 0
2006-11-03 09:54:00 Sent 126 bytes
2006-11-03 09:54:00 Read 16384 bytes
2006-11-03 09:54:00 Read 32767 bytes
2006-11-03 09:54:00 Read 1 bytes
2006-11-03 09:54:00 Read 32767 bytes
2006-11-03 09:54:00 Read 32767 bytes
2006-11-03 09:54:00 Read 24578 bytes
2006-11-03 09:54:00 Read 32767 bytes
2006-11-03 09:54:00 Read 32767 bytes
2006-11-03 09:54:00 Read 24578 bytes
2006-11-03 09:54:00 Read 16503 bytes
2006-11-03 09:54:00 Closing connection