[hobbit] hosts still disappear

[Bruce Lysik]

I just realized I left out the actual backtrace.  Here's one I just ran
again:

0xb7584416 in semop () from /lib/tls/libc.so.6
(gdb) bt
#0  0xb7584416 in semop () from /lib/tls/libc.so.6
#1  0x08049f27 in posttochannel (channel=0xbfffc740, 
    channelmarker=0x80538b2 "status", 
    msg=0x8101868 "status uberbox.sys green Tue Jan 18 16:51:44 2005
\nSystem Description  NetApp Release 6.5.2: Sun Jul 25 10:56:02 PDT
2004\n\nObject ID", ' ' <repeats 11 times>,
".1.3.6.1.4.1.789.2.1\n\nUptime", ' ' <repeats 14 times>, "143 days,
10:42"..., sender=0x8053bbc "hobbitd", hostname=0x81017f0 "uberbox", 
    log=0x8101800, readymsg=0x0) at hobbitd.c:390
#2  0x0804ab22 in handle_status (
    msg=0x8101868 "status uberbox.sys green Tue Jan 18 16:51:44 2005
\nSystem Description  NetApp Release 6.5.2: Sun Jul 25 10:56:02 PDT
2004\n\nObject ID", ' ' <repeats 11 times>,
".1.3.6.1.4.1.789.2.1\n\nUptime", ' ' <repeats 14 times>, "143 days,
10:42"..., sender=0x8053bbc "hobbitd", hostname=0x81017f0 "uberbox", 
    testname=0x7e6bf <Address 0x7e6bf out of bounds>, log=0x8101800, 
    newcolor=3) at hobbitd.c:830
#3  0x0804df9a in check_purple_status () at hobbitd.c:2001
#4  0x0804ebff in main (argc=7, argv=0xbfffcb14) at hobbitd.c:2359

--
Bruce Z. Lysik  <blysik at shutterfly.com>
Operations Engineer


> -----Original Message-----
> From: Bruce Lysik 
> Sent: Wednesday, January 19, 2005 9:35 AM
> To: hobbit at hswn.dk
> Subject: RE: [hobbit] hosts still disappear
> 
> 
> 
> Correct, beta-5, plus the patch.  
> 
> Okay, hobbitd isn't responding at all.  hobbitd.chk hasn't 
> been updated
> since yesterday, and the hobbitlog command doesn't return anything.
> 
> Strangely, no pid file was located in /var/log/hobbit.  However, a ps
> grepping for hobbit showed this:
> 
> bbuser   32296 32295  0 Jan18 ?        00:00:00 hobbitd
> --restart=/opt/bb/server/tmp/hobbitd.chk
> --checkpoint-file=/opt/bb/server/tmp/hobbitd.chk 
> --checkpoint-interval=600
> --purple-conn=conn --log=/var/log/hobbit/hobbitd.log
> --admin-senders=127.0.0.1 172.16.190.2
> 
> I used gdb and attached to that pid, resulting in:
> 
> (gdb) attach 32296
> Attaching to process 32296
> Reading symbols from /opt/bb/server/bin/hobbitd...done.
> Using host libthread_db library "/lib/tls/libthread_db.so.1".
> Reading symbols from /lib/tls/libc.so.6...done.
> Loaded symbols for /lib/tls/libc.so.6
> Reading symbols from /lib/ld-linux.so.2...done.
> Loaded symbols for /lib/ld-linux.so.2
> 0xb7584416 in semop () from /lib/tls/libc.so.6
> (gdb) 
> 
> 
> 
> -----Original Message-----
> From: Henrik Stoerner [mailto:henrik at hswn.dk]
> Sent: Tue 1/18/2005 10:58 PM
> To: hobbit at hswn.dk
> Subject: Re: [hobbit] hosts still disappear
>  
> On Tue, Jan 18, 2005 at 03:17:51PM -0800, Bruce Lysik wrote:
> > 
> > Well, I spoke too soon about hosts disappearing.  I'm still 
> seeing the
> same
> > problem.  Is there anything I can do to help in debugging this?
> 
> This is with the beta-5 version PLUS the patch I posted yesterday ?
> 
> First, check if the hobbit daemon is still responding at all. You
> can see if the ~server/tmp/hobbitd.chk file is being updated - it
> should refresh every 10 minutes. You can also try this: If
> 
>  ~/server/bin/bb 127.0.0.1 "hobbitdlog some,server,name.conn"
> 
> responds with a status log, then it is responding. If not, proceed:
> 
> 
> Next, I want to figure out what the hobbitd proces is doing. You'll
> need the gdb (GNU debugger) for that. Login as the hobbit user, Find
> the process ID of hobbitd - it's recorded in
> /var/log/hobbit/hobbitd.pid, or use "ps" -  then:
> 
>   gdb ~/server/bin/hobbitd
>   [gdb messages]
>   gdb> attach <hobbitd-processid>
>   gdb> bt
> 
> Let me know what this shows.
> 
> 
> Henrik
> 
> To unsubscribe from the hobbit list, send an e-mail to
> hobbit-unsubscribe at hswn.dk
> 
> 
> 
>