[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [hobbit] moving hobbit from Ubuntu to Debian



This probably isn't going to be much help with this problem, but Henrik might find the feed back of some interest.

I recently rebooted a Sun server. Later the hobbit client status for the server went purple and I noticed I was getting these errors in the hobbitclient.log on that server:
2006-05-25 06:58:38 Whoops ! bb failed to send message - timeout
2006-05-25 07:03:43 Whoops ! bb failed to send message - timeout
2006-05-25 07:08:47 Whoops ! bb failed to send message - timeout


However I was getting one status from the client. It was from an add-on application called SE Tool Kit. It displayed the following status:

green    Disks  2%busy No activity
green    Nets  No activity
green    NFS   No client NFS/RPC activity
green    Swap  There is a lot of unused swap space
green    RAM   86%free RAM available
green    Kmem  No worries, mate
green    CPU   CPU idling
green    Mutex No worries, mate
green    DNLC  No worries, mate
green    Inode No activity
yellow   TCP   High retransmissions, check TCP patch level

From this we were able to determine there was a network problem. Our LAN group got involved and resolved the retransmission errors from their side.

Getting back to the original Whoops error. In our case it seems the hobbit client was not getting thru because of the high retransmissions. As for why the addon messages got thru and the rest of the hobbit client messages didn't I will leave that up to Henrik for his thoughts on it.


For those who may be interested in setting up SE Tool Kit with the hobbit client for Sun. You can get it from here http://www.sunfreeware.com/setoolkit.html. There is an example file included in the package called bigbrother.se. I renamed it to hobbit.se, updated the path for calling bin/bb and changed the BBHOST variable to BBDISP. Add an entry to clientlaunch.cfg:
[se]
ENVFILE $HOBBITCLIENTHOME/etc/hobbitclient.cfg
CMD /opt/RICHPse/bin/se /opt/hobbit/client/hobbit.se
LOGFILE $HOBBITCLIENTHOME/logs/hobbitclient.log



We also run Orca(http://www.orcaware.com/orca/). It's a performance and trend analysis package. It also uses the SE Tool kit and creates detailed graphs. Here is an what it looks like.
http://www.orcaware.com/orca/docs/orcallator.html#TCP_retransmission_duplicate_received_percentage


With how fast Hobbit is growing we may someday be able to drop the Orca package.

John

Dan Simoes wrote:
I appreciate the help.  Running the debug you sent, it seems that the box
cannot connect to itself on 1984 using its public IP.
I will see if I can remove the firewall for now (it's not iptables, it's
something the provider gives you).
Thanks again.

On 5/31/06, Henrik Stoerner <henrik (at) hswn.dk> wrote:


One more thing before I go to sleep :-)

Check your hobbitserver.cfg - if you have BBDISP set the same as
BBSERVERIP. That *is* the default, but you might have changed it.
It should be
   BBDISP="$BBSERVERIP"
in hobbitserver.cfg.


Regards, Henrik

On Thu, Jun 01, 2006 at 12:04:55AM +0200, Henrik Stoerner wrote:
> Login as the hobbit user, then run
> ./server/bin/bbcmd bbgen --debug | less
> A bit into that log is where bbgen tries to fetch the current status
> from the Hobbit daemon. It looks like this:
>
> 2006-06-01 00:02:24 load_state()
> 2006-06-01 00:02:24 Transport setup is:
> 2006-06-01 00:02:24 bbdportnumber = 1984
> 2006-06-01 00:02:24 bbdispproxyhost = NONE
> 2006-06-01 00:02:24 bbdispproxyport = 0
> 2006-06-01 00:02:24 Recipient listed as '127.0.0.1'
> 2006-06-01 00:02:24 Standard BB protocol on port 1984
> 2006-06-01 00:02:24 Will connect to address 127.0.0.1 port 1984
> 2006-06-01 00:02:24 Connect status is 0
> 2006-06-01 00:02:24 Sent 118 bytes
> 2006-06-01 00:02:24 Read 3364 bytes
>
> Does it report the right IP-address as the 'Recipient' ?
>
> Henrik
>
>
> On Wed, May 31, 2006 at 03:00:19PM -0700, Dan Simoes wrote:
> > Telnet to localhost as well as 127.0.0.1 work, yes.
> > The hobbitdboard returns a lot of data.
> >
> > I've literally been looking at this for hours, I don't know what else
is
> > wrong.
> > I even did a completely fresh install, dropped in my bb-hosts, same
results.
> > I know bb-hosts is OK since it's working elsewhere (I did change the
bbd
> > line of course).
> >
> > On 5/31/06, Henrik Stoerner <henrik (at) hswn.dk> wrote:
> > >
> > >Can you telnet to 127.0.0.1 port 1984 ? If that times out, then you
have
> > >a seriously heavy firewall setup. Something like
> > > iptables -I INPUT 1 -i lo -j ACCEPT
> > > iptables -I OUTPUT 1 -i lo -j ACCEPT
> > >should cancel out any paranoid firewall setups locally on the box,
for
> > >the loopback-interface.
> > >
> > >Can you do a "~hobbit/server/bin/bb 127.0.0.1 hobbitdboard" and get
any
> > >data back ?
> > >
> > >If you're running all of the network tests locally, I think you can
> > >get away with setting
> > > BBSERVERIP="127.0.0.1"
> > >in your hobbitserver.cfg, and let all of the communications happen on
> > >the loopback-interface. Of course, for clients to work you'll need to
> > >get port 1984 opened up for access from the clients. But loopback
should
> > >be ok for just getting the Hobbit server up and running.
> > >
> > >
> > >Henrik
> > >
> > >
> > >On Wed, May 31, 2006 at 02:48:33PM -0700, Dan Simoes wrote:
> > >> I thought of that, eventually, and corrected the error. Same
problem.
> > >> localhost is responding on 1984, but the site's firewall is
blocking
> > >1984.
> > >> I don't have any clients connecting at the moment, so I don't
imagine
> > >this
> > >> is a problem, but thought I would mention it.
> > >>
> > >> On 5/31/06, Henrik Stoerner <henrik (at) hswn.dk> wrote:
> > >> >
> > >> >Check the IP-adresses for the Hobbit server in the
hobbitserver.cfg
> > >> >file. From your logs, it seems that the query to fetch the current
> > >total
> > >> >status from the Hobbit daemon goes to the wrong server (the old
IP,
> > >> >probably), so there is no response and therefore the webpages dont
get
> > >> >any hosts listed.
> > >> >
> > >> >
> > >> >Henrik
> > >> >
> > >> >On Wed, May 31, 2006 at 02:41:25PM -0700, Dan Simoes wrote:
> > >> >> I got an account on a new virtual host (debian 3.1 on
virtuozzo), and
> > >> >> dropped in a tar of my hobbit dir.
> > >> >> I figured I could just edit bb-hosts and be up and running in
> > >minutes.
> > >> >> Hours later, I'm still not up.
> > >> >>
> > >> >> OK, so I was missing fping, and rrdtool. Hobbit did not like
the
> > >older
> > >> >> rrdtool 1.X under debian stable, so I switch to testing,
upgraded,
> > >and
> > >> >> recompiled and reinstalled hobbit.
> > >> >> It's still not working, even with a simplified bb-hosts.
> > >> >>
> > >> >> hobbit runs, but the webpage output is devoid of any host data.
> > >> >> I can't figure out what else could be wrong, this should have
been a
> > >> >very
> > >> >> simple port.
> > >> >>
> > >> >> Logs show:
> > >> >>
> > >> >> bb--display:
> > >> >> 2006-05-31 17:30:22 Whoops ! bb failed to send message - timeout
> > >> >> 2006-05-31 17:30:22 hobbitd status-board not available
> > >> >> 2006-05-31 17:30:22 WARNING: Runtime 92 longer than BBSLEEP (60)
> > >> >> 2006-05-31 17:31:09 Whoops ! bb failed to send message - timeout
> > >> >>
> > >> >> bb-network:
> > >> >> 2006-05-31 17:30:27 Whoops ! bb failed to send message - timeout
> > >> >> 2006-05-31 17:30:27 WARNING: Runtime 140 longer than time limit
(120)
> > >> >> 2006-05-31 17:31:14 Whoops ! bb failed to send message - timeout
> > >> >>
> > >> >>
> > >> >> hobbitclient.log:
> > >> >> 2006-05-31 17:32:47 Whoops ! bb failed to send message - timeout
> > >> >> error parsing /proc/net/snmp: Success
> > >> >> 2006-05-31 17:34:47 Whoops ! bb failed to send message - timeout
> > >> >> error parsing /proc/net/snmp: Success
> > >> >
> > >> >--
> > >> >Henrik Storner
> > >> >
> > >> >To unsubscribe from the hobbit list, send an e-mail to
> > >> >hobbit-unsubscribe (at) hswn.dk
> > >> >
> > >> >
> > >> >
> > >
> > >--
> > >Henrik Storner
> > >
> > >To unsubscribe from the hobbit list, send an e-mail to
> > >hobbit-unsubscribe (at) hswn.dk
> > >
> > >
> > >
>
> --
> Henrik Storner
>
> To unsubscribe from the hobbit list, send an e-mail to
> hobbit-unsubscribe (at) hswn.dk
>
>


--
Henrik Storner

To unsubscribe from the hobbit list, send an e-mail to
hobbit-unsubscribe (at) hswn.dk