[hobbit] Serious hobbit problem, client data truncated by server

[Gore, David W (David)]

I have a very serious hobbit problem.  Our hobbit has been working very
well for more than a year.  I have rolled back some config files,
bb-hosts, client-local.cfg, and hobbit-clients.cfg, on the hopes one of
them may have a typo causing hobbit to act erratically.  Unfortunately,
no luck.

So what is the problem?  The client sends, msg.<host>.txt, as some of
you may know, and you can see this file on the server or web page via
the 'Client data' link.  Unfortunately, the hobbit server is truncating
the '[ps]' listing which means you lose all the other entries after
'[ps]' and now you are also going to start alarming on missing
processes.

Alarming and paging out the on-call on missing processes in the middle
of the night and creating bogus tickets is very bad.  There isn't too
much in the logs, but we do have something.

Starting on June 02 we got this in bb-display.log:

2007-06-04 12:21:07 Whoops ! bb failed to send message - timeout
2007-06-04 12:21:07 hobbitd status-board not available
2007-06-04 14:21:47 Whoops ! bb failed to send message - timeout
2007-06-04 14:21:47 hobbitd status-board not available
2007-06-04 15:02:02 Whoops ! bb failed to send message - timeout
2007-06-04 15:02:02 hobbitd status-board not available

Any ideas?  Henrik?  


Oh and of course the message size is more than adequate to handle the
data. We have many hosts that send 2-3 times more data on average and
nothing has changed on the client.
 

David