[hobbit] moving hobbit from Ubuntu to Debian

John Glowacki johng at idttechnology.com
Fri Jun 2 06:41:47 CEST 2006


This probably isn't going to be much help with this problem, but Henrik 
might find the feed back of some interest.

I recently rebooted a Sun server. Later the hobbit client status for the 
server went purple and I noticed I was getting these errors in the 
hobbitclient.log on that server:
2006-05-25 06:58:38 Whoops ! bb failed to send message - timeout
2006-05-25 07:03:43 Whoops ! bb failed to send message - timeout
2006-05-25 07:08:47 Whoops ! bb failed to send message - timeout

However I was getting one status from the client. It was from an add-on 
application called SE Tool Kit. It displayed the following status:

green    Disks  2%busy No activity
green    Nets  No activity
green    NFS   No client NFS/RPC activity
green    Swap  There is a lot of unused swap space
green    RAM   86%free RAM available
green    Kmem  No worries, mate
green    CPU   CPU idling
green    Mutex No worries, mate
green    DNLC  No worries, mate
green    Inode No activity
yellow   TCP   High retransmissions, check TCP patch level

 From this we were able to determine there was a network problem. Our 
LAN group got involved and resolved the retransmission errors from their 
side.

Getting back to the original Whoops error. In our case it seems the 
hobbit client was not getting thru because of the high retransmissions. 
As for why the addon messages got thru and the rest of the hobbit client 
messages didn't I will leave that up to Henrik for his thoughts on it.


For those who may be interested in setting up SE Tool Kit with the 
hobbit client for Sun. You can get it from here 
http://www.sunfreeware.com/setoolkit.html. There is an example file 
included in the package called bigbrother.se. I renamed it to hobbit.se, 
updated the path for calling bin/bb and changed the BBHOST variable to 
BBDISP. Add an entry to clientlaunch.cfg:
[se]
         ENVFILE $HOBBITCLIENTHOME/etc/hobbitclient.cfg
         CMD /opt/RICHPse/bin/se /opt/hobbit/client/hobbit.se
         LOGFILE $HOBBITCLIENTHOME/logs/hobbitclient.log


We also run Orca(http://www.orcaware.com/orca/). It's a performance and 
trend analysis package. It also uses the SE Tool kit and creates 
detailed graphs. Here is an what it looks like.
http://www.orcaware.com/orca/docs/orcallator.html#TCP_retransmission_duplicate_received_percentage

With how fast Hobbit is growing we may someday be able to drop the Orca 
package.

John

Dan Simoes wrote:
> I appreciate the help.  Running the debug you sent, it seems that the box
> cannot connect to itself on 1984 using its public IP.
> I will see if I can remove the firewall for now (it's not iptables, it's
> something the provider gives you).
> Thanks again.
> 
> On 5/31/06, Henrik Stoerner <henrik at hswn.dk> wrote:
> 
>>
>> One more thing before I go to sleep :-)
>>
>> Check your hobbitserver.cfg - if you have BBDISP set the same as
>> BBSERVERIP. That *is* the default, but you might have changed it.
>> It should be
>>    BBDISP="$BBSERVERIP"
>> in hobbitserver.cfg.
>>
>>
>> Regards,
>> Henrik
>>
>> On Thu, Jun 01, 2006 at 12:04:55AM +0200, Henrik Stoerner wrote:
>> > Login as the hobbit user, then run
>> >    ./server/bin/bbcmd bbgen --debug | less
>> > A bit into that log is where bbgen tries to fetch the current status
>> > from the Hobbit daemon. It looks like this:
>> >
>> > 2006-06-01 00:02:24 load_state()
>> > 2006-06-01 00:02:24 Transport setup is:
>> > 2006-06-01 00:02:24 bbdportnumber = 1984
>> > 2006-06-01 00:02:24 bbdispproxyhost = NONE
>> > 2006-06-01 00:02:24 bbdispproxyport = 0
>> > 2006-06-01 00:02:24 Recipient listed as '127.0.0.1'
>> > 2006-06-01 00:02:24 Standard BB protocol on port 1984
>> > 2006-06-01 00:02:24 Will connect to address 127.0.0.1 port 1984
>> > 2006-06-01 00:02:24 Connect status is 0
>> > 2006-06-01 00:02:24 Sent 118 bytes
>> > 2006-06-01 00:02:24 Read 3364 bytes
>> >
>> > Does it report the right IP-address as the 'Recipient' ?
>> >
>> > Henrik
>> >
>> >
>> > On Wed, May 31, 2006 at 03:00:19PM -0700, Dan Simoes wrote:
>> > > Telnet to localhost as well as 127.0.0.1 work, yes.
>> > > The hobbitdboard returns a lot of data.
>> > >
>> > > I've literally been looking at this for hours, I don't know what else
>> is
>> > > wrong.
>> > > I even did a completely fresh install, dropped in my bb-hosts, same
>> results.
>> > > I know bb-hosts is OK since it's working elsewhere (I did change the
>> bbd
>> > > line of course).
>> > >
>> > > On 5/31/06, Henrik Stoerner <henrik at hswn.dk> wrote:
>> > > >
>> > > >Can you telnet to 127.0.0.1 port 1984 ? If that times out, then you
>> have
>> > > >a seriously heavy firewall setup. Something like
>> > > >   iptables -I INPUT 1 -i lo -j ACCEPT
>> > > >   iptables -I OUTPUT 1 -i lo -j ACCEPT
>> > > >should cancel out any paranoid firewall setups locally on the box,
>> for
>> > > >the loopback-interface.
>> > > >
>> > > >Can you do a "~hobbit/server/bin/bb 127.0.0.1 hobbitdboard" and get
>> any
>> > > >data back ?
>> > > >
>> > > >If you're running all of the network tests locally, I think you can
>> > > >get away with setting
>> > > >   BBSERVERIP="127.0.0.1"
>> > > >in your hobbitserver.cfg, and let all of the communications 
>> happen on
>> > > >the loopback-interface. Of course, for clients to work you'll 
>> need to
>> > > >get port 1984 opened up for access from the clients. But loopback
>> should
>> > > >be ok for just getting the Hobbit server up and running.
>> > > >
>> > > >
>> > > >Henrik
>> > > >
>> > > >
>> > > >On Wed, May 31, 2006 at 02:48:33PM -0700, Dan Simoes wrote:
>> > > >> I thought of that, eventually, and corrected the error.  Same
>> problem.
>> > > >> localhost is responding on 1984, but the site's firewall is
>> blocking
>> > > >1984.
>> > > >> I don't have any clients connecting at the moment, so I don't
>> imagine
>> > > >this
>> > > >> is a problem, but thought I would mention it.
>> > > >>
>> > > >> On 5/31/06, Henrik Stoerner <henrik at hswn.dk> wrote:
>> > > >> >
>> > > >> >Check the IP-adresses for the Hobbit server in the
>> hobbitserver.cfg
>> > > >> >file. From your logs, it seems that the query to fetch the 
>> current
>> > > >total
>> > > >> >status from the Hobbit daemon goes to the wrong server (the old
>> IP,
>> > > >> >probably), so there is no response and therefore the webpages 
>> dont
>> get
>> > > >> >any hosts listed.
>> > > >> >
>> > > >> >
>> > > >> >Henrik
>> > > >> >
>> > > >> >On Wed, May 31, 2006 at 02:41:25PM -0700, Dan Simoes wrote:
>> > > >> >> I got an account on a new virtual host (debian 3.1 on
>> virtuozzo), and
>> > > >> >> dropped in a tar of my hobbit dir.
>> > > >> >> I figured I could just edit bb-hosts and be up and running in
>> > > >minutes.
>> > > >> >> Hours later, I'm still not up.
>> > > >> >>
>> > > >> >> OK, so I was missing fping, and rrdtool.  Hobbit did not like
>> the
>> > > >older
>> > > >> >> rrdtool 1.X under debian stable, so I switch to testing,
>> upgraded,
>> > > >and
>> > > >> >> recompiled and reinstalled hobbit.
>> > > >> >> It's still not working, even with a simplified bb-hosts.
>> > > >> >>
>> > > >> >> hobbit runs, but the webpage output is devoid of any host data.
>> > > >> >> I can't figure out what else could be wrong, this should have
>> been a
>> > > >> >very
>> > > >> >> simple port.
>> > > >> >>
>> > > >> >> Logs show:
>> > > >> >>
>> > > >> >> bb--display:
>> > > >> >> 2006-05-31 17:30:22 Whoops ! bb failed to send message - 
>> timeout
>> > > >> >> 2006-05-31 17:30:22 hobbitd status-board not available
>> > > >> >> 2006-05-31 17:30:22 WARNING: Runtime 92 longer than BBSLEEP 
>> (60)
>> > > >> >> 2006-05-31 17:31:09 Whoops ! bb failed to send message - 
>> timeout
>> > > >> >>
>> > > >> >> bb-network:
>> > > >> >> 2006-05-31 17:30:27 Whoops ! bb failed to send message - 
>> timeout
>> > > >> >> 2006-05-31 17:30:27 WARNING: Runtime 140 longer than time limit
>> (120)
>> > > >> >> 2006-05-31 17:31:14 Whoops ! bb failed to send message - 
>> timeout
>> > > >> >>
>> > > >> >>
>> > > >> >> hobbitclient.log:
>> > > >> >> 2006-05-31 17:32:47 Whoops ! bb failed to send message - 
>> timeout
>> > > >> >> error parsing /proc/net/snmp: Success
>> > > >> >> 2006-05-31 17:34:47 Whoops ! bb failed to send message - 
>> timeout
>> > > >> >> error parsing /proc/net/snmp: Success
>> > > >> >
>> > > >> >--
>> > > >> >Henrik Storner
>> > > >> >
>> > > >> >To unsubscribe from the hobbit list, send an e-mail to
>> > > >> >hobbit-unsubscribe at hswn.dk
>> > > >> >
>> > > >> >
>> > > >> >
>> > > >
>> > > >--
>> > > >Henrik Storner
>> > > >
>> > > >To unsubscribe from the hobbit list, send an e-mail to
>> > > >hobbit-unsubscribe at hswn.dk
>> > > >
>> > > >
>> > > >
>> >
>> > --
>> > Henrik Storner
>> >
>> > To unsubscribe from the hobbit list, send an e-mail to
>> > hobbit-unsubscribe at hswn.dk
>> >
>> >
>>
>> -- 
>> Henrik Storner
>>
>> To unsubscribe from the hobbit list, send an e-mail to
>> hobbit-unsubscribe at hswn.dk
>>
>>
>>
> 



More information about the Xymon mailing list