Re: [hobbit] hobbitd_larrd is crashing

["Larry Barber" <lebarber (at) gmail.com>]

I loaded p1, and hobbitd_rrd is still dumping, the stack trace looks like:

#0  0x00dfe60a in do_lookup_versioned () from /lib/ld-linux.so.2
#1  0x00dfd776 in _dl_lookup_versioned_symbol_internal () from /lib/ld-
linux.so.2
#2  0x00e01473 in fixup () from /lib/ld-linux.so.2
#3  0x00e01330 in _dl_runtime_resolve () from /lib/ld-linux.so.2
#4  0x08054c6d in sigsegv_handler (signum=11) at sig.c:51
#5  <signal handler called>
#6  0x00dfe3da in do_lookup () from /lib/ld-linux.so.2
#7  0x00dfd103 in _dl_lookup_symbol_internal () from /lib/ld-linux.so.2
#8  0x00e0140f in fixup () from /lib/ld-linux.so.2
#9  0x00e01330 in _dl_runtime_resolve () from /lib/ld-linux.so.2
#10 0x0804a91f in create_and_update_rrd (hostname=0xb755d037
"stellent_pre-prod_v-ip",
   fn=0x805f6e0
"tcp.http.https:,,pws.tc.sc.egov.usda.gov,siteminderagent,dmsforms,login_banner.fcc?TYPE=33554433&REALMOID=06-d38f4375-a8bd-4190-b6f9-3c77f0901647&GUID=&SMAUTHREASON=0&METHOD=GET&SMAGENTNAME=$SM$hIspF3"...,
creparams=0x805e5c0, template=0x9cf6b20 "sec") at do_rrd.c:143
#11 0x0804f294 in do_net_rrd (hostname=0xb755d037 "stellent_pre-prod_v-ip",
testname=0xb755d04e "http",
   msg=0xb755d07c "status stellent_pre-prod_v-ip.http green Fri Jun  9
16:16:31 2006: OK ; OK\n\n&green
https://pws.tc.sc.egov.usda.gov/siteminderagent/dmsforms/login_banner.fcc?TYPE=33554433&REALMOID=06-d38f4375-a8bd-419";...,
tstamp=1149887818) at rrd/do_net.c:48
#12 0x0805024a in update_rrd (hostname=0xb755d037 "stellent_pre-prod_v-ip",
testname=0xb755d04e "http",
   msg=0xb755d07c "status stellent_pre-prod_v-ip.http green Fri Jun  9
16:16:31 2006: OK ; OK\n\n&green
https://pws.tc.sc.egov.usda.gov/siteminderagent/dmsforms/login_banner.fcc?TYPE=33554433&REALMOID=06-d38f4375-a8bd-419";...,
tstamp=1149887818, sender=0x1ca3f <Address 0x1ca3f out of bounds>,
ldef=0x1ca3f) at do_rrd.c:291
#13 0x08049cf0 in main (argc=117311, argv=0xbfff8324) at hobbitd_rrd.c:199

larrd-status.log looks like:

...

2006-06-09 15:45:24 Our child has failed and will not talk to us: Channel
status, PID 22591
2006-06-09 15:45:24 Worker process died with exit code 139, terminating
2006-06-09 15:56:03 2006-06-09 15:56:03 Worker process died with exit code
139, terminating
2006-06-09 15:57:03 2006-06-09 15:57:03 Worker process died with exit code
139, terminating
2006-06-09 15:57:24 Worker process died with exit code 139, terminating
2006-06-09 15:57:24 Our child has failed and will not talk to us: Channel
status, PID 25060
2006-06-09 15:57:24 Worker process died with exit code 139, terminating
2006-06-09 15:59:24 2006-06-09 15:59:24 Worker process died with exit code
139, terminating
2006-06-09 15:59:24 Worker process died with exit code 139, terminating
2006-06-09 16:09:26 Worker process died with exit code 139, terminating
2006-06-09 16:13:01 2006-06-09 16:13:01 Worker process died with exit code
139, terminating
2006-06-09 16:13:02 Worker process died with exit code 139, terminating
2006-06-09 16:14:56 2006-06-09 16:14:56 Worker process died with exit code
139, terminating
2006-06-09 16:14:57 Worker process died with exit code 139, terminating
2006-06-09 16:16:58 2006-06-09 16:16:58 Worker process died with exit code
139, terminating
2006-06-09 16:16:58 Worker process died with exit code 139, terminating


It just started doing this today, I can't think of anything that I have done
that could cause it.

Thanks,
Larry Barber


On 6/9/06, Henrik Stoerner <henrik (at) hswn.dk> wrote:
> On Fri, Jun 09, 2006 at 01:30:39PM -0500, Larry Barber wrote:
> > For some reason hobbitd_larrd has started crashing on my main production
> > server. Larrd-status.log has messages like this in it:
> >
> > 2006-06-09 12:42:39 Worker process died with exit code 139, terminating
> >
> > Loading the core file into gdb and executin "backtrace" yields "No
> stack".
> > Any ideas what's going on? I'm running Hobbit 4.1.2.rc1 on a RedHat ES3
> box.
>
> 4.1.2-rc1 is pretty old (almost one year).
>
> I can think of several problems that might cause this, but my first
> suggestion would be to at least upgrade to the 4.1.2p1 release that is
> the current production-release.
>
> From a testing perspective I'd like you to try out the 4.2 beta release
> that went out early this week, but I fully understand if you would
> rather not run the beta-version on a production system.
>
>
> Regards,
> Henrik
>
>
> To unsubscribe from the hobbit list, send an e-mail to
> hobbit-unsubscribe (at) hswn.dk