[hobbit] moving hobbit from Ubuntu to Debian
John Glowacki
johng at idttechnology.com
Fri Jun 2 06:41:47 CEST 2006
This probably isn't going to be much help with this problem, but Henrik
might find the feed back of some interest.
I recently rebooted a Sun server. Later the hobbit client status for the
server went purple and I noticed I was getting these errors in the
hobbitclient.log on that server:
2006-05-25 06:58:38 Whoops ! bb failed to send message - timeout
2006-05-25 07:03:43 Whoops ! bb failed to send message - timeout
2006-05-25 07:08:47 Whoops ! bb failed to send message - timeout
However I was getting one status from the client. It was from an add-on
application called SE Tool Kit. It displayed the following status:
green Disks 2%busy No activity
green Nets No activity
green NFS No client NFS/RPC activity
green Swap There is a lot of unused swap space
green RAM 86%free RAM available
green Kmem No worries, mate
green CPU CPU idling
green Mutex No worries, mate
green DNLC No worries, mate
green Inode No activity
yellow TCP High retransmissions, check TCP patch level
From this we were able to determine there was a network problem. Our
LAN group got involved and resolved the retransmission errors from their
side.
Getting back to the original Whoops error. In our case it seems the
hobbit client was not getting thru because of the high retransmissions.
As for why the addon messages got thru and the rest of the hobbit client
messages didn't I will leave that up to Henrik for his thoughts on it.
For those who may be interested in setting up SE Tool Kit with the
hobbit client for Sun. You can get it from here
http://www.sunfreeware.com/setoolkit.html. There is an example file
included in the package called bigbrother.se. I renamed it to hobbit.se,
updated the path for calling bin/bb and changed the BBHOST variable to
BBDISP. Add an entry to clientlaunch.cfg:
[se]
ENVFILE $HOBBITCLIENTHOME/etc/hobbitclient.cfg
CMD /opt/RICHPse/bin/se /opt/hobbit/client/hobbit.se
LOGFILE $HOBBITCLIENTHOME/logs/hobbitclient.log
We also run Orca(http://www.orcaware.com/orca/). It's a performance and
trend analysis package. It also uses the SE Tool kit and creates
detailed graphs. Here is an what it looks like.
http://www.orcaware.com/orca/docs/orcallator.html#TCP_retransmission_duplicate_received_percentage
With how fast Hobbit is growing we may someday be able to drop the Orca
package.
John
Dan Simoes wrote:
> I appreciate the help. Running the debug you sent, it seems that the box
> cannot connect to itself on 1984 using its public IP.
> I will see if I can remove the firewall for now (it's not iptables, it's
> something the provider gives you).
> Thanks again.
>
> On 5/31/06, Henrik Stoerner <henrik at hswn.dk> wrote:
>
>>
>> One more thing before I go to sleep :-)
>>
>> Check your hobbitserver.cfg - if you have BBDISP set the same as
>> BBSERVERIP. That *is* the default, but you might have changed it.
>> It should be
>> BBDISP="$BBSERVERIP"
>> in hobbitserver.cfg.
>>
>>
>> Regards,
>> Henrik
>>
>> On Thu, Jun 01, 2006 at 12:04:55AM +0200, Henrik Stoerner wrote:
>> > Login as the hobbit user, then run
>> > ./server/bin/bbcmd bbgen --debug | less
>> > A bit into that log is where bbgen tries to fetch the current status
>> > from the Hobbit daemon. It looks like this:
>> >
>> > 2006-06-01 00:02:24 load_state()
>> > 2006-06-01 00:02:24 Transport setup is:
>> > 2006-06-01 00:02:24 bbdportnumber = 1984
>> > 2006-06-01 00:02:24 bbdispproxyhost = NONE
>> > 2006-06-01 00:02:24 bbdispproxyport = 0
>> > 2006-06-01 00:02:24 Recipient listed as '127.0.0.1'
>> > 2006-06-01 00:02:24 Standard BB protocol on port 1984
>> > 2006-06-01 00:02:24 Will connect to address 127.0.0.1 port 1984
>> > 2006-06-01 00:02:24 Connect status is 0
>> > 2006-06-01 00:02:24 Sent 118 bytes
>> > 2006-06-01 00:02:24 Read 3364 bytes
>> >
>> > Does it report the right IP-address as the 'Recipient' ?
>> >
>> > Henrik
>> >
>> >
>> > On Wed, May 31, 2006 at 03:00:19PM -0700, Dan Simoes wrote:
>> > > Telnet to localhost as well as 127.0.0.1 work, yes.
>> > > The hobbitdboard returns a lot of data.
>> > >
>> > > I've literally been looking at this for hours, I don't know what else
>> is
>> > > wrong.
>> > > I even did a completely fresh install, dropped in my bb-hosts, same
>> results.
>> > > I know bb-hosts is OK since it's working elsewhere (I did change the
>> bbd
>> > > line of course).
>> > >
>> > > On 5/31/06, Henrik Stoerner <henrik at hswn.dk> wrote:
>> > > >
>> > > >Can you telnet to 127.0.0.1 port 1984 ? If that times out, then you
>> have
>> > > >a seriously heavy firewall setup. Something like
>> > > > iptables -I INPUT 1 -i lo -j ACCEPT
>> > > > iptables -I OUTPUT 1 -i lo -j ACCEPT
>> > > >should cancel out any paranoid firewall setups locally on the box,
>> for
>> > > >the loopback-interface.
>> > > >
>> > > >Can you do a "~hobbit/server/bin/bb 127.0.0.1 hobbitdboard" and get
>> any
>> > > >data back ?
>> > > >
>> > > >If you're running all of the network tests locally, I think you can
>> > > >get away with setting
>> > > > BBSERVERIP="127.0.0.1"
>> > > >in your hobbitserver.cfg, and let all of the communications
>> happen on
>> > > >the loopback-interface. Of course, for clients to work you'll
>> need to
>> > > >get port 1984 opened up for access from the clients. But loopback
>> should
>> > > >be ok for just getting the Hobbit server up and running.
>> > > >
>> > > >
>> > > >Henrik
>> > > >
>> > > >
>> > > >On Wed, May 31, 2006 at 02:48:33PM -0700, Dan Simoes wrote:
>> > > >> I thought of that, eventually, and corrected the error. Same
>> problem.
>> > > >> localhost is responding on 1984, but the site's firewall is
>> blocking
>> > > >1984.
>> > > >> I don't have any clients connecting at the moment, so I don't
>> imagine
>> > > >this
>> > > >> is a problem, but thought I would mention it.
>> > > >>
>> > > >> On 5/31/06, Henrik Stoerner <henrik at hswn.dk> wrote:
>> > > >> >
>> > > >> >Check the IP-adresses for the Hobbit server in the
>> hobbitserver.cfg
>> > > >> >file. From your logs, it seems that the query to fetch the
>> current
>> > > >total
>> > > >> >status from the Hobbit daemon goes to the wrong server (the old
>> IP,
>> > > >> >probably), so there is no response and therefore the webpages
>> dont
>> get
>> > > >> >any hosts listed.
>> > > >> >
>> > > >> >
>> > > >> >Henrik
>> > > >> >
>> > > >> >On Wed, May 31, 2006 at 02:41:25PM -0700, Dan Simoes wrote:
>> > > >> >> I got an account on a new virtual host (debian 3.1 on
>> virtuozzo), and
>> > > >> >> dropped in a tar of my hobbit dir.
>> > > >> >> I figured I could just edit bb-hosts and be up and running in
>> > > >minutes.
>> > > >> >> Hours later, I'm still not up.
>> > > >> >>
>> > > >> >> OK, so I was missing fping, and rrdtool. Hobbit did not like
>> the
>> > > >older
>> > > >> >> rrdtool 1.X under debian stable, so I switch to testing,
>> upgraded,
>> > > >and
>> > > >> >> recompiled and reinstalled hobbit.
>> > > >> >> It's still not working, even with a simplified bb-hosts.
>> > > >> >>
>> > > >> >> hobbit runs, but the webpage output is devoid of any host data.
>> > > >> >> I can't figure out what else could be wrong, this should have
>> been a
>> > > >> >very
>> > > >> >> simple port.
>> > > >> >>
>> > > >> >> Logs show:
>> > > >> >>
>> > > >> >> bb--display:
>> > > >> >> 2006-05-31 17:30:22 Whoops ! bb failed to send message -
>> timeout
>> > > >> >> 2006-05-31 17:30:22 hobbitd status-board not available
>> > > >> >> 2006-05-31 17:30:22 WARNING: Runtime 92 longer than BBSLEEP
>> (60)
>> > > >> >> 2006-05-31 17:31:09 Whoops ! bb failed to send message -
>> timeout
>> > > >> >>
>> > > >> >> bb-network:
>> > > >> >> 2006-05-31 17:30:27 Whoops ! bb failed to send message -
>> timeout
>> > > >> >> 2006-05-31 17:30:27 WARNING: Runtime 140 longer than time limit
>> (120)
>> > > >> >> 2006-05-31 17:31:14 Whoops ! bb failed to send message -
>> timeout
>> > > >> >>
>> > > >> >>
>> > > >> >> hobbitclient.log:
>> > > >> >> 2006-05-31 17:32:47 Whoops ! bb failed to send message -
>> timeout
>> > > >> >> error parsing /proc/net/snmp: Success
>> > > >> >> 2006-05-31 17:34:47 Whoops ! bb failed to send message -
>> timeout
>> > > >> >> error parsing /proc/net/snmp: Success
>> > > >> >
>> > > >> >--
>> > > >> >Henrik Storner
>> > > >> >
>> > > >> >To unsubscribe from the hobbit list, send an e-mail to
>> > > >> >hobbit-unsubscribe at hswn.dk
>> > > >> >
>> > > >> >
>> > > >> >
>> > > >
>> > > >--
>> > > >Henrik Storner
>> > > >
>> > > >To unsubscribe from the hobbit list, send an e-mail to
>> > > >hobbit-unsubscribe at hswn.dk
>> > > >
>> > > >
>> > > >
>> >
>> > --
>> > Henrik Storner
>> >
>> > To unsubscribe from the hobbit list, send an e-mail to
>> > hobbit-unsubscribe at hswn.dk
>> >
>> >
>>
>> --
>> Henrik Storner
>>
>> To unsubscribe from the hobbit list, send an e-mail to
>> hobbit-unsubscribe at hswn.dk
>>
>>
>>
>
More information about the Xymon
mailing list