Re: [Devmon] [hobbit] Re: Devmon causing core dumps

[Buchan Milne <bgmilne (at) staff.telkomsa.net>]

On Thursday, 19 November 2009 18:49:28 mario andre wrote:
> Hi,
>
> Bunchan.
>
> I'm getting hobbitd_rrd crash. I'm running 4.2.3 RC1

Any reason not to use the final release of 4.2.3?

> with your do_devmon.c
> revision 154.
> I figured that the core file are repeating for the same equipments. cisco
> 6513 and cisco 4507 for if_load.

How did you determine this?

> THis could be because of the number of interfaces? What do you think?

Unlikely. Some 6509's and 7613's I was monitoring had about the same number or 
more ports and didn't have problems.

> Thanks in advance,
>
> Mario.
>
> Cisco 4507:
>
> Alarming on
> (Gi1/1,Gi1/2,Gi2/1,Gi2/2,Gi3/1,Gi3/2,Gi3/3,Gi3/4,Gi3/5,Gi3/6,Gi3/7)
> Alarming on (Gi3/8,Gi3/9,Gi3/10,Gi3/11,Gi3/12,Gi3/13,Gi3/14,Gi3/15)
> Alarming on (Gi3/16,Gi3/17,Gi3/18,Gi3/19,Gi3/20,Gi3/21,Gi3/22)
> Alarming on (Gi3/23,Gi3/24,Gi3/25,Gi3/26,Gi3/27,Gi3/28,Gi3/29)
> Alarming on (Gi3/30,Gi3/31,Gi3/32,Gi3/33,Gi3/34,Gi3/35,Gi3/36)
> Alarming on (Gi3/37,Gi3/38,Gi3/39,Gi3/40,Gi3/41,Gi3/42,Gi3/43)
> Alarming on (Gi3/44,Gi3/45,Gi3/46,Gi3/47,Gi3/48,Gi4/1,Gi4/2,Gi4/3)
> Alarming on (Gi4/4,Gi4/5,Gi4/6,Gi4/7,Gi4/8,Gi4/9,Gi4/10,Gi4/11)
> Alarming on (Gi4/12,Gi4/13,Gi4/14,Gi4/15,Gi4/16,Gi4/17,Gi4/18)
> Alarming on (Gi4/19,Gi4/20,Gi4/21,Gi4/22,Gi4/23,Gi4/24,Gi4/25)
> Alarming on (Gi4/26,Gi4/27,Gi4/28,Gi4/29,Gi4/30,Gi4/31,Gi4/32)
> Alarming on (Gi4/33,Gi4/34,Gi4/35,Gi4/36,Gi4/37,Gi4/38,Gi4/39)
> Alarming on (Gi4/40,Gi4/41,Gi4/42,Gi4/43,Gi4/44,Gi4/45,Gi4/46)
> Alarming on (Gi4/47,Gi4/48,Gi5/1,Gi5/2,Gi5/3,Gi5/4,Gi5/5,Gi5/6)
> Alarming on (Gi5/7,Gi5/8,Gi5/9,Gi5/10,Gi5/11,Gi5/12,Gi5/13,Gi5/14)
> Alarming on (Gi5/15,Gi5/16,Gi5/17,Gi5/18,Gi5/19,Gi5/20,Gi5/21)
> Alarming on (Gi5/22,Gi5/23,Gi5/24,Gi5/25,Gi5/26,Gi5/27,Gi5/28)
> Alarming on (Gi5/29,Gi5/30,Gi5/31,Gi5/32,Gi5/33,Gi5/34,Gi5/35)
> Alarming on (Gi5/36,Gi5/37,Gi5/38,Gi5/39,Gi5/40,Gi5/41,Gi5/42)
> Alarming on (Gi5/43,Gi5/44,Gi5/45,Gi5/46,Gi5/47,Gi5/48,Gi6/1)
> Alarming on (Gi6/2,Gi6/3,Gi6/4,Gi6/5,Gi6/6,Gi6/7,Gi6/8,Gi6/9)
> Alarming on (Gi6/10,Gi6/11,Gi6/12,Gi6/13,Gi6/14,Gi6/15,Gi6/16)
> Alarming on (Gi6/17,Gi6/18,Gi6/19,Gi6/20,Gi6/21,Gi6/22,Gi6/23)
> Alarming on (Gi6/24,Gi6/25,Gi6/26,Gi6/27,Gi6/28,Gi6/29,Gi6/30)
> Alarming on (Gi6/31,Gi6/32,Gi6/33,Gi6/34,Gi6/35,Gi6/36,Gi6/37)
> Alarming on (Gi6/38,Gi6/39,Gi6/40,Gi6/41,Gi6/42,Gi6/43,Gi6/44)
> Alarming on (Gi6/45,Gi6/46,Gi6/47,Gi6/48)
>
>
>
> And Cisco 6513:
>
> Input load:  yellow=75%, red=95%
> Output load: yellow=75%, red=95%
> Alarming on
> (Gi1/1,Gi1/2,Gi2/1,Gi2/2,Gi3/1,Gi3/2,Gi3/3,Gi3/4,Gi3/5,Gi3/6,Gi3/7)
> Alarming on (Gi3/8,Gi3/9,Gi3/10,Gi3/11,Gi3/12,Gi3/13,Gi3/14,Gi3/15)
> Alarming on (Gi3/16,Gi3/17,Gi3/18,Gi3/19,Gi3/20,Gi3/21,Gi3/22)
> Alarming on (Gi3/23,Gi3/24,Gi3/25,Gi3/26,Gi3/27,Gi3/28,Gi3/29)
> Alarming on (Gi3/30,Gi3/31,Gi3/32,Gi3/33,Gi3/34,Gi3/35,Gi3/36)
> Alarming on (Gi3/37,Gi3/38,Gi3/39,Gi3/40,Gi3/41,Gi3/42,Gi3/43)
> Alarming on (Gi3/44,Gi3/45,Gi3/46,Gi3/47,Gi3/48,Fa6/1,Fa6/2,Fa6/3)
> Alarming on (Fa6/4,Fa6/5,Fa6/6,Fa6/7,Fa6/8,Fa6/9,Fa6/10,Fa6/11)
> Alarming on (Fa6/12,Fa6/13,Fa6/14,Fa6/15,Fa6/16,Fa6/17,Fa6/18)
> Alarming on (Fa6/19,Fa6/20,Fa6/21,Fa6/22,Fa6/23,Fa6/24,Fa6/25)
> Alarming on (Fa6/26,Fa6/27,Fa6/28,Fa6/29,Fa6/30,Fa6/31,Fa6/32)
> Alarming on (Fa6/33,Fa6/34,Fa6/35,Fa6/36,Fa6/37,Fa6/38,Fa6/39)
> Alarming on (Fa6/40,Fa6/41,Fa6/42,Fa6/43,Fa6/44,Fa6/45,Fa6/46)
> Alarming on (Fa6/47,Fa6/48,Fa9/1,Fa9/2,Fa9/3,Fa9/4,Fa9/5,Fa9/6)
> Alarming on (Fa9/7,Fa9/8,Fa9/9,Fa9/10,Fa9/11,Fa9/12,Fa9/13,Fa9/14)
> Alarming on (Fa9/15,Fa9/16,Fa9/17,Fa9/18,Fa9/19,Fa9/20,Fa9/21)
> Alarming on (Fa9/22,Fa9/23,Fa9/24,Fa9/25,Fa9/26,Fa9/27,Fa9/28)
> Alarming on (Fa9/29,Fa9/30,Fa9/31,Fa9/32,Fa9/33,Fa9/34,Fa9/35)
> Alarming on (Fa9/36,Fa9/37,Fa9/38,Fa9/39,Fa9/40,Fa9/41,Fa9/42)
> Alarming on (Fa9/43,Fa9/44,Fa9/45,Fa9/46,Fa9/47,Fa9/48,Fa10/1)
> Alarming on (Fa10/2,Fa10/3,Fa10/4,Fa10/5,Fa10/6,Fa10/7,Fa10/8)
> Alarming on (Fa10/9,Fa10/10,Fa10/11,Fa10/12,Fa10/13,Fa10/14,Fa10/15)
> Alarming on (Fa10/16,Fa10/17,Fa10/18,Fa10/19,Fa10/20,Fa10/21)
> Alarming on (Fa10/22,Fa10/23,Fa10/24,Fa10/25,Fa10/26,Fa10/27)
> Alarming on (Fa10/28,Fa10/29,Fa10/30,Fa10/31,Fa10/32,Fa10/33)
> Alarming on (Fa10/34,Fa10/35,Fa10/36,Fa10/37,Fa10/38,Fa10/39)
> Alarming on (Fa10/40,Fa10/41,Fa10/42,Fa10/43,Fa10/44,Fa10/45)
> Alarming on (Fa10/46,Fa10/47,Fa10/48,Gi11/1,Gi11/2,Gi11/3,Gi11/4)
> Alarming on (Gi11/5,Gi11/6,Gi11/7,Gi11/8,Gi11/9,Gi11/10,Gi11/11)
> Alarming on (Gi11/12,Gi11/13,Gi11/14,Gi11/15,Gi11/16,Gi12/1,Gi12/2)
> Alarming on (Gi12/3,Gi12/4,Gi12/5,Gi12/6,Gi12/7,Gi12/8,Gi12/9)
> Alarming on (Gi12/10,Gi12/11,Gi12/12,Gi12/13,Gi12/14,Gi12/15)
> Alarming on (Gi12/16,Gi13/1,Gi13/2,Gi13/3,Gi13/4,Gi13/5,Gi13/6)
> Alarming on (Gi13/7,Gi13/8,Gi13/9,Gi13/10,Gi13/11,Gi13/12,Gi13/13)
> Alarming on (Gi13/14,Gi13/15,Gi13/16,EO0/0,Po1,Po2,Po7,Po8,Gi5/1)
> Alarming on (Gi5/2,Gi5/3,Gi5/4,Gi5/5,Gi5/6,Gi5/7,Gi5/8,Gi5/9)
> Alarming on (Gi5/10,Gi5/11,Gi5/12,Gi5/13,Gi5/14,Gi5/15,Gi5/16)
> Alarming on (Gi5/17,Gi5/18,Gi5/19,Gi5/20,Gi5/21,Gi5/22,Gi5/23)
> Alarming on (Gi5/24,Gi5/25,Gi5/26,Gi5/27,Gi5/28,Gi5/29,Gi5/30)
> Alarming on (Gi5/31,Gi5/32,Gi5/33,Gi5/34,Gi5/35,Gi5/36,Gi5/37)
> Alarming on (Gi5/38,Gi5/39,Gi5/40,Gi5/41,Gi5/42,Gi5/43,Gi5/44)
> Alarming on (Gi5/45,Gi5/46,Gi5/47,Gi5/48,Po10,Po30,Po50)

There are no funny names here, which is what would have caused problems in 
previous versions of the devmon collector (but rev 154 shouldn't).

Can you provide a backtrace from one of the core files? It will provide at 
least the devmon test name that triggered the core, and possibly information 
that would help to fix the issue. You can do that with something like:

gdb /path/to/xymon-4.2.3RC1/hobbitd/hobbitd_rrd -c /path/to/core.xxxx

Where the path to hobbitd above is the hobbitd in the source directory where 
you built the hobbitd in question. Then, in the gdb prompt, type 'bt full'

It would also help more to provide the output of something like:

bb localhost 'hobbitdlog hostname.if_load'. This would allow me to test with 
the exact data your server is getting.

Regards,
Buchan