hobbitd_rrd crashing over and over.

[Whitney Green <wgreen (at) bridge2solutions.com>]

I am running xymon 4.3.0-0 beta 2 on a RHEL 5.4 x86_64 arch.

Every 5mins or so, I get an alert that hobbit_rrd has crashed, and I get a ton of core files.

# uname -a
Linux xymon 2.6.18-164.el5xen #1 SMP Tue Aug 18 15:59:52 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux

# rpm -qa |grep rrd
rrdtool-1.2.23-1.el5.rf
rrdtool-devel-1.2.23-1.el5.rf
perl-rrdtool-1.2.23-1.el5.rf



[root (at) xymon server]# file tmp/core.11912
tmp/core.11912: ELF 64-bit LSB core file AMD x86-64, version 1 (SYSV), SVR4-style, from 'hobbitd_rrd'

Here is the backtrace:

# gdb bin/hobbitd_rrd tmp/core.11912
GNU gdb Fedora (6.8-37.el5)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...
Reading symbols from /usr/lib64/librrd.so.2...done.
Loaded symbols for /usr/lib64/librrd.so.2
Reading symbols from /lib64/libpcre.so.0...done.
Loaded symbols for /lib64/libpcre.so.0
Reading symbols from /lib64/librt.so.1...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libc.so.6...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /usr/lib64/libfreetype.so.6...done.
Loaded symbols for /usr/lib64/libfreetype.so.6
Reading symbols from /usr/lib64/libpng12.so.0...done.
Loaded symbols for /usr/lib64/libpng12.so.0
Reading symbols from /usr/lib64/libz.so.1...done.
Loaded symbols for /usr/lib64/libz.so.1
Reading symbols from /usr/lib64/libart_lgpl_2.so.2...done.
Loaded symbols for /usr/lib64/libart_lgpl_2.so.2
Reading symbols from /lib64/libm.so.6...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libpthread.so.0...done.
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Core was generated by `hobbitd_rrd --rrddir=/opt/hobbit/data/rrd'.
Program terminated with signal 6, Aborted.
[New process 11912]
#0  0x000000398ec30265 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x000000398ec30265 in raise () from /lib64/libc.so.6
#1  0x000000398ec31d10 in abort () from /lib64/libc.so.6
#2  0x00000000004201c3 in sigsegv_handler (signum=<value optimized out>) at sig.c:57
#3  <signal handler called>
#4  0x000000385dc2a823 in write_RRA_row (rrd=0x7fffea126180, rra_idx=3, rra_current=0x7fffea126218,
    CDP_scratch_idx=<value optimized out>, rrd_file=0x1a3e84f0, pcdp_summary=0x0,
    rra_time=0x7fffea126208, rrd_mmaped_file=0x2b2a8173a000) at /usr/include/bits/string3.h:51
#5  0x000000385dc2c3e3 in _rrd_update (
    filename=0x63b6c0 "/opt/hobbit/data/rrd/xymon/vmstat.rrd",
    tmplt=<value optimized out>, argc=9, argv=0x7fffea1262d0, pcdp_summary=0x0) at rrd_update.c:1230
#6  0x000000385dc2d207 in rrd_update (argc=13, argv=0x7fffea1262b0) at rrd_update.c:183
#7  0x000000000040497b in flush_cached_updates (cacheitem=0x1a3a3580, newdata=<value optimized out>)
    at do_rrd.c:223
#8  0x00000000004065b9 in create_and_update_rrd (hostname=0x2b2a8152bfb7 "xymon",
    testname=0x2b2a8152bfc8 "vmstat", classname=<value optimized out>,
    pagepaths=<value optimized out>, creparams=0x1a3da170, template=0x1a3a2bb0) at do_rrd.c:393
#9  0x000000000040d8b2 in do_vmstat_rrd (hostname=0x2b2a8152bfb7 "xymon",
    testname=0x2b2a8152bfc8 "vmstat", classname=0x2b2a8152bfcf "linux", pagepaths=0x426002 "",
    msg=<value optimized out>, tstamp=<value optimized out>) at rrd/do_vmstat.c:436
#10 0x0000000000410885 in update_rrd (hostname=0x2b2a8152bfb7 "xymon",
    testname=0x2b2a8152bfc8 "vmstat", msg=0x2b2a8152bfd6 "data xymon.vmstat\nlinux\n 0",
    tstamp=1257351440, sender=<value optimized out>, ldef=<value optimized out>,
    classname=0x2b2a8152bfcf "linux", pagepaths=0x426002 "") at do_rrd.c:591
#11 0x0000000000403017 in main (argc=<value optimized out>, argv=0x7fffea129908)
    at hobbitd_rrd.c:366

I have done fresh recompiles of multiple versions of xymon,  used different versions of the rrd packages, and this error still constantly pops up.

I get a ton of errors in rrd-data.log:

2009-11-04 11:17:24 Peer at 0.0.0.0:0 failed: Broken pipe
2009-11-04 11:17:24 Peer not up, flushing message queue
2009-11-04 11:27:26 Peer at 0.0.0.0:0 failed: Broken pipe
2009-11-04 11:27:26 Peer not up, flushing message queue
2009-11-04 11:42:33 Peer at 0.0.0.0:0 failed: Broken pipe
2009-11-04 11:42:33 Peer not up, flushing message queue
2009-11-04 11:46:08 Peer at 0.0.0.0:0 failed: Broken pipe
2009-11-04 11:46:08 Peer not up, flushing message queue

I've honestly never had this problem before, and I'm pretty baffled.