[hobbit] Serious hobbit problem, client data truncated by server

[Gore, David W (David)]

> -----Original Message-----
> From: Gore, David W (David) 
> Sent: Monday, June 04, 2007 17:52
> To: hobbit at hswn.dk
> Subject: [hobbit] Serious hobbit problem, client data 
> truncated by server
> 
> I have a very serious hobbit problem.  Our hobbit has been 
> working very
> well for more than a year.  I have rolled back some config files,
> bb-hosts, client-local.cfg, and hobbit-clients.cfg, on the 
> hopes one of
> them may have a typo causing hobbit to act erratically.  
> Unfortunately,
> no luck.
> 
> So what is the problem?  The client sends, msg.<host>.txt, as some of
> you may know, and you can see this file on the server or web page via
> the 'Client data' link.  Unfortunately, the hobbit server is 
> truncating
> the '[ps]' listing which means you lose all the other entries after
> '[ps]' and now you are also going to start alarming on missing
> processes.
> 
> Alarming and paging out the on-call on missing processes in the middle
> of the night and creating bogus tickets is very bad.  There isn't too
> much in the logs, but we do have something.
> 
> Starting on June 02 we got this in bb-display.log:
> 
> 2007-06-04 12:21:07 Whoops ! bb failed to send message - timeout
> 2007-06-04 12:21:07 hobbitd status-board not available
> 2007-06-04 14:21:47 Whoops ! bb failed to send message - timeout
> 2007-06-04 14:21:47 hobbitd status-board not available
> 2007-06-04 15:02:02 Whoops ! bb failed to send message - timeout
> 2007-06-04 15:02:02 hobbitd status-board not available
> 
> Any ideas?  Henrik?  
> 
> 
> Oh and of course the message size is more than adequate to handle the
> data. We have many hosts that send 2-3 times more data on average and
> nothing has changed on the client.
>  

Missing info from my first post: 

Dual hobbit servers running Hobbit 4.2.0 w/allinone patch on Fedora Core
5 on Dell Optiplex GX620 dual core P4 3.2GHz w/1G of memory each


Here is the solution:

Here is what my netstat looks like on the problem host (hobbit1):

[root at hobbit1 ~]# netstat -in
Kernel Interface table
Iface       MTU Met    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP
TX-OVR Flg
eth0       1500   0 39784213 17711963    383      0 41615768      0
0      0 BMRU
lo        16436   0 10299557      0      0      0 10299557      0      0
0 LRU

Obviously not good.  This host WAS our primary hobbit server with a
10Mb/s half-duplex connection.  Our secondary hobbit server runs at
100Mb/s full-duplex.

Additionally, I had this setting in hobbitserver.cfg:

BBDISP="0.0.0.0"               # IP of a single hobbit/bbd server
BBDISPLAYS="hobbit1 hobbit2"   # IP of multiple hobbit/bbd servers. If
used, BBDISP must be 0.0.0.0

I am not sure what that does with parallel servers, but it does appear
to tie them together and when one is down perhaps this causes problems.
This setting was probably made more than a year ago, and perhaps it just
took time to catch up with us combined with the slow network connection.
I really am not sure why you would do what I did or what it's effect is
but I have changed both servers to this setting:

BBDISP="$BBSERVERIP"           # IP of a single hobbit/bbd server
BBDISPLAYS=""                  # IP of multiple hobbit/bbd servers. If
used, BBDISP must be 0.0.0.0

I have also moved the primary server to the 100Mb/s full-duplex
connection and have plans to move the secondary to a 100Mb/s full-duplex
connection.

So far everything appears to be running fine.  Perhaps now when I do
multi-host clientupdates it will work as expected.  This slow connection
could easily explain why some messages from the client were being
truncated on the server and why clientupdates also caused problems.  I
should have also found the problem a lot sooner.

Thanks go to Henrik and Mike Rowell for helping.

~David