Devmon causing core dumps

[Buchan Milne]

On Friday 31 October 2008 05:51:42 Everett, Vernon wrote:
> Hi all
>
> Devmon was causing the hobbitd_rrd module to crash and burn.
> Now this could be a bug, but it could also be a PEBKAC. I am hoping
> somebody can assist either way.
>
> I added a Cisco 2851 to Hobbit, using devmon.
> Now here is the possible PEBKAC
> Since Devmon doesn't have templates for the 2851, I used the template for
> the Cisco 2811. (Network guru told me they are pretty much the same, except
> for a few extra bells and whistles on the 2851.)
>
> The data for the device started appearing in Hobbit, and all looked good.
> Devmon even created the rrd files for the new Cisco device.
>
> However, the hobbitd_rrd module started core dumping, and the Hobbit server
> page started displaying red for hobbitd_rrd with the crash detected
> message. See core data below.
> Took the new Cisco device out of Hobbit, and cores stopped, and life was
> good again.
>
> Is there a significant enough difference between the 2851 and the 2811 to
> cause this, or are we looking at a genuine bug?

Real bug. I see it on the temperature tests on a new IOS.

> I am leaning towards a bug,
> because even if the collected data was complete rubbish, should it cause
> the module to core?
>
> Regards
>      Vernon
>
> My Linux guy reckons this is the important stuff from the core.
> uname -a
> Linux las006 2.6.18-92.1.1.el5 #1 SMP Thu May 22 09:01:47 EDT 2008 x86_64
> x86_64 x86_64 GNU/Linux cat /etc/redhat-release Red Hat Enterprise Linux
> Client release 5.2 (Tikanga)
>
> gdb -c core.8550 /usr/lib/hobbit/server/bin/hobbitd_rrd
> GNU gdb Red Hat Linux (6.5-37.el5_2.1rh) Copyright (C) 2006 Free Software
> Foundation, Inc. GDB is free software, covered by the GNU General Public
> License, and you are welcome to change it and/or distribute copies of it
> under certain conditions. Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu"...Using host
> libthread_db library "/lib64/libthread_db.so.1".
>
> Reading symbols from /usr/lib64/librrd.so.2...done.
> Loaded symbols for /usr/lib64/librrd.so.2 Reading symbols from
> /usr/lib64/libpng12.so.0...done. Loaded symbols for
> /usr/lib64/libpng12.so.0 Reading symbols from /lib64/libpcre.so.0...done.
> Loaded symbols for /lib64/libpcre.so.0
> Reading symbols from /lib64/libc.so.6...done.
> Loaded symbols for /lib64/libc.so.6
> Reading symbols from /usr/lib64/libfreetype.so.6...done.
> Loaded symbols for /usr/lib64/libfreetype.so.6 Reading symbols from
> /usr/lib64/libz.so.1...done. Loaded symbols for /usr/lib64/libz.so.1
> Reading symbols from /usr/lib64/libart_lgpl_2.so.2...done.
> Loaded symbols for /usr/lib64/libart_lgpl_2.so.2 Reading symbols from
> /lib64/libm.so.6...done. Loaded symbols for /lib64/libm.so.6
> Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
> Loaded symbols for /lib64/ld-linux-x86-64.so.2 Core was generated by
> `hobbitd_rrd --rrddir=/var/lib/hobbit/rrd --debug'. Program terminated with
> signal 6, Aborted.
> #0  0x0000003db7a30155 in raise () from /lib64/libc.so.6
> (gdb) bt
> #0  0x0000003db7a30155 in raise () from /lib64/libc.so.6
> #1  0x0000003db7a31bf0 in abort () from /lib64/libc.so.6
> #2  0x00000000004119f3 in sigsegv_handler (signum=<value optimized out>) at
> sig.c:57 #3  <signal handler called>
> #4  0x0000003db7a77ac0 in strcat () from /lib64/libc.so.6
> #5  0x000000000040462a in do_devmon_rrd (hostname=0x2ada311e2806
> "PERIR205", testname=0x2ada311e280f "if_load", msg=<value optimized out>,
> tstamp=<value optimized out>) at rrd/do_devmon.c:87
> #6  0x000000000040b656 in update_rrd (hostname=0x2ada311e2806 "PERIR205",
> testname=0x2ada311e280f "if_load", msg=0x2ada311e2842 "status
> PERIR205.if_load green Fri Oct 31 10:31:39 2008", tstamp=1225416699,
> sender=<value optimized out>, ldef=0xfeffffffffffff00) at do_rrd.c:372 #7 
> 0x000000000040261d in main (argc=<value optimized out>,
> argv=0x7fff7a088318) at hobbitd_rrd.c:153 (gdb)


Could you show the Devmon RRD section of the message for the if_load test on 
the PERIR205 host? I can confirm the cause, and maybe offer a workaround.

I am actually (constantly) reproducing the issue on my workstation against the 
new IOS that can trigger this, I have a workaround in place in production, and 
was hoping to get around to fixing this next week.

Regards,
Buchan