core dumps after modifying allevents file

Tue Jan 22 12:44:15 CET 2008

Hi All,

My allevents file was nearing the 2GB mark but was 95% full of lines from two custom tests that changed state every 30 seconds or so. These were removed manually as trim history could not be used as we would loose valuable historic data. allevents file cannot exceed 2GB.

The file was cat'ed out with a grep -v of the unwated tests, completed fine and now new allevents file is 141mb.

Hobbit was stopped before the process started, a ps showed no hobbit processes running, above change was made, permissions were checked in all events to ensure they were the same, hobbit was then started.

about 500 hosts are monitored with most of them running client side custom scripts, very shortly after startup core files were generated and hobbit either crashed or stopped itself.

Below is the debug output from the core files and relevant log line entries round this time. The only way to get hobbit started back up and the hobbit status page displayed, was to firewall off the server so that all status messages were ignored (iptables used), stop hobbit, ensure all processes are gone and started again, waited a few minutes so that hobbit could do what it wanted, then the iptables rule was removed. After 10 mins all returned to normal and no more core files have been generated.

Not sure if whilst hobbit is starting up, if it receives too many status updates, or any during startup it causes it to core dump as it is not ready?

Any help would be greatly appreciated. 

*****************
CORE FILES
****************

-rw-------    1 hobbit   hobbit        23M Jan 22 09:36 core.27740
-rw-------    1 hobbit   hobbit        23M Jan 22 09:37 core.28187
-rw-------    1 hobbit   hobbit        23M Jan 22 09:38 core.28385

**********************
DEBUG OUTPUT
*********************

# gdb ../bin/hobbitd core.27740
GNU gdb Red Hat Linux (6.3.0.0-1.138.el3rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db library "/lib/tls/libthread_db.so.1".

Core was generated by `hobbitd --pidfile=/var/log/hobbit/hobbitd.pid --restart=/var/hobbit/server/tmp/'.
Program terminated with signal 6, Aborted.
Reading symbols from /lib/libpcre.so.0...done.
Loaded symbols for /lib/libpcre.so.0
Reading symbols from /lib/tls/libc.so.6...done.
Loaded symbols for /lib/tls/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
#0  0x00434eff in raise () from /lib/tls/libc.so.6
(gdb) bt
#0  0x00434eff in raise () from /lib/tls/libc.so.6
#1  0x00436705 in abort () from /lib/tls/libc.so.6
#2  0x0805a006 in sigsegv_handler (signum=11) at sig.c:57
#3  <signal handler called>
#4  0x0048701c in mempcpy () from /lib/tls/libc.so.6
#5  0x0047a6a2 in _IO_default_xsputn_internal () from /lib/tls/libc.so.6
#6  0x004535c7 in vfprintf () from /lib/tls/libc.so.6
#7  0x0046f4ec in vsprintf () from /lib/tls/libc.so.6
#8  0x0045a0dd in sprintf () from /lib/tls/libc.so.6
#9  0x0804df40 in generate_outbuf (outbuf=0xbffee4c4, outpos=0xbffee4c8, outsz=0xbffee4cc, hwalk=0x89793b0, lwalk=0x897b350, acklevel=-1)
    at hobbitd.c:2227
#10 0x0805074f in do_message (msg=0x8df5d10, origin=0x805c4fc "") at hobbitd.c:2848
#11 0x08053472 in main (argc=8, argv=0xbfff18e4) at hobbitd.c:4377

# gdb /var/hobbit/server/bin/hobbitd /var/hobbit/server/tmp/core.28187
GNU gdb Red Hat Linux (6.3.0.0-1.138.el3rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db library "/lib/tls/libthread_db.so.1".

Core was generated by `hobbitd --pidfile=/var/log/hobbit/hobbitd.pid --restart=/var/hobbit/server/tmp/'.
Program terminated with signal 6, Aborted.
Reading symbols from /lib/libpcre.so.0...done.
Loaded symbols for /lib/libpcre.so.0
Reading symbols from /lib/tls/libc.so.6...done.
Loaded symbols for /lib/tls/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
#0  0x0093aeff in raise () from /lib/tls/libc.so.6
(gdb) bt
#0  0x0093aeff in raise () from /lib/tls/libc.so.6
#1  0x0093c705 in abort () from /lib/tls/libc.so.6
#2  0x0805a006 in sigsegv_handler (signum=11) at sig.c:57
#3  <signal handler called>
#4  0x0098d01c in mempcpy () from /lib/tls/libc.so.6
#5  0x009806a2 in _IO_default_xsputn_internal () from /lib/tls/libc.so.6
#6  0x009595c7 in vfprintf () from /lib/tls/libc.so.6
#7  0x009754ec in vsprintf () from /lib/tls/libc.so.6
#8  0x009600dd in sprintf () from /lib/tls/libc.so.6
#9  0x0804df40 in generate_outbuf (outbuf=0xbfff0f74, outpos=0xbfff0f78, outsz=0xbfff0f7c, hwalk=0x9352838, lwalk=0x93557a0, acklevel=-1)
    at hobbitd.c:2227
#10 0x0805074f in do_message (msg=0x9c27bb0, origin=0x805c4fc "") at hobbitd.c:2848
#11 0x08053472 in main (argc=8, argv=0xbfff4394) at hobbitd.c:4377
(gdb) quit

# gdb /var/hobbit/server/bin/hobbitd /var/hobbit/server/tmp/core.28385
GNU gdb Red Hat Linux (6.3.0.0-1.138.el3rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db library "/lib/tls/libthread_db.so.1".

Core was generated by `hobbitd --pidfile=/var/log/hobbit/hobbitd.pid --restart=/var/hobbit/server/tmp/'.
Program terminated with signal 6, Aborted.
Reading symbols from /lib/libpcre.so.0...done.
Loaded symbols for /lib/libpcre.so.0
Reading symbols from /lib/tls/libc.so.6...done.
Loaded symbols for /lib/tls/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
#0  0x00abfeff in raise () from /lib/tls/libc.so.6
(gdb) bt
#0  0x00abfeff in raise () from /lib/tls/libc.so.6
#1  0x00ac1705 in abort () from /lib/tls/libc.so.6
#2  0x0805a006 in sigsegv_handler (signum=11) at sig.c:57
#3  <signal handler called>
#4  0x00b1201c in mempcpy () from /lib/tls/libc.so.6
#5  0x00b056a2 in _IO_default_xsputn_internal () from /lib/tls/libc.so.6
#6  0x00ade5c7 in vfprintf () from /lib/tls/libc.so.6
#7  0x00afa4ec in vsprintf () from /lib/tls/libc.so.6
#8  0x00ae50dd in sprintf () from /lib/tls/libc.so.6
#9  0x0804df40 in generate_outbuf (outbuf=0xbfff3774, outpos=0xbfff3778, outsz=0xbfff377c, hwalk=0x9c863b0, lwalk=0x9c88350, acklevel=-1)
    at hobbitd.c:2227
#10 0x0805074f in do_message (msg=0xa678100, origin=0x805c4fc "") at hobbitd.c:2848
#11 0x08053472 in main (argc=8, argv=0xbfff6b94) at hobbitd.c:4377
(gdb) quit

***************
LOG FILES
**************
hobbitlaunch.log:2008-01-22 09:36:05 hobbitlaunch starting
hobbitlaunch.log:2008-01-22 09:36:05 Loading tasklist configuration from /var/hobbit/server/etc/hobbitlaunch.cfg
hobbitlaunch.log:2008-01-22 09:36:05 Loading hostnames
hobbitlaunch.log:2008-01-22 09:36:05 Loading saved state
hobbitlaunch.log:2008-01-22 09:36:06 Setting up network listener on 0.0.0.0:1984
hobbitlaunch.log:2008-01-22 09:36:06 Setting up signal handlers
hobbitlaunch.log:2008-01-22 09:36:06 Setting up hobbitd channels
hobbitlaunch.log:2008-01-22 09:36:06 Setting up logfiles
hobbitlaunch.log:2008-01-22 09:36:10 Task hobbitd terminated by signal 6
hobbitlaunch.log:2008-01-22 09:36:10 Loading hostnames
hobbitlaunch.log:2008-01-22 09:36:10 Task bbdisplay terminated by signal 15
hobbitlaunch.log:2008-01-22 09:36:10 Task bbnet terminated by signal 15
hobbitlaunch.log:2008-01-22 09:36:10 Task hobbitclient terminated by signal 15
hobbitlaunch.log:2008-01-22 09:36:10 Loading saved state
hobbitlaunch.log:2008-01-22 09:36:11 Setting up network listener on 0.0.0.0:1984
hobbitlaunch.log:2008-01-22 09:36:11 Setting up signal handlers
hobbitlaunch.log:2008-01-22 09:36:11 Setting up hobbitd channels
hobbitlaunch.log:2008-01-22 09:36:11 Setting up logfiles
hobbitlaunch.log:2008-01-22 09:37:12 Task hobbitd terminated by signal 6
hobbitlaunch.log:2008-01-22 09:37:12 Task bbdisplay terminated by signal 15
hobbitlaunch.log:2008-01-22 09:37:12 Loading hostnames
hobbitlaunch.log:2008-01-22 09:37:12 Loading saved state
hobbitlaunch.log:2008-01-22 09:37:12 Setting up network listener on 0.0.0.0:1984
hobbitlaunch.log:2008-01-22 09:37:12 Setting up signal handlers
hobbitlaunch.log:2008-01-22 09:37:12 Setting up hobbitd channels
hobbitlaunch.log:2008-01-22 09:37:12 Setting up logfiles
hobbitlaunch.log:2008-01-22 09:38:14 Task hobbitd terminated by signal 6
hobbitlaunch.log:2008-01-22 09:38:14 Task bbdisplay terminated by signal 15
hobbitlaunch.log:2008-01-22 09:45:27 hobbitlaunch starting
hobbitlaunch.log:2008-01-22 09:45:27 Loading tasklist configuration from /var/hobbit/server/etc/hobbitlaunch.cfg
hobbitlaunch.log:2008-01-22 09:45:27 Loading hostnames
hobbitlaunch.log:2008-01-22 09:45:27 Loading saved state
hobbitlaunch.log:2008-01-22 09:45:27 Setting up network listener on 0.0.0.0:1984
hobbitlaunch.log:2008-01-22 09:45:27 Setting up signal handlers
hobbitlaunch.log:2008-01-22 09:45:27 Setting up hobbitd channels
hobbitlaunch.log:2008-01-22 09:45:27 Setting up logfiles

bb-display.log:2008-01-22 09:37:12 2008-01-22 09:38:14 2008-01-22 09:45:47 Whoops ! bb failed to send message - timeout

bb-display.log:2008-01-22 09:45:47 hobbitd status-board not available

bb-display.log:2008-01-22 09:46:34 Whoops ! bb failed to send message - timeout

bb-display.log:2008-01-22 09:47:21 Whoops ! bb failed to send message - timeout

bb-display.log:2008-01-22 09:47:21 hobbitd status-board not available

bb-display.log:2008-01-22 09:48:08 Whoops ! bb failed to send message - timeout

bb-display.log:2008-01-22 09:48:55 Whoops ! bb failed to send message - timeout

bb-display.log:2008-01-22 09:48:55 hobbitd status-board not available

hobbitclient.log:2008-01-22 09:46:25 Whoops ! bb failed to send message - timeout

hobbitclient.log:2008-01-22 09:51:26 Whoops ! bb failed to send message - timeout

hobbitclient.log:2008-01-22 09:56:26 Whoops ! bb failed to send message - timeout

hobbitd.log:2008-01-22 09:27:47 BOARDBUSY locked at 1, GETNCNT is 0, GETPID is 0, 0 clients

hobbitd.log:2008-01-22 09:36:06 Setup complete

hobbitd.log:2008-01-22 09:36:11 Setup complete

hobbitd.log:2008-01-22 09:37:12 Setup complete

hobbitd.log:2008-01-22 09:45:27 Setup complete

Thanks
 --

 Regards

 Dominic

      __________________________________________________________
Sent from Yahoo! Mail - a smarter inbox http://uk.mail.yahoo.com