RE: Scheduled disable causes crash?

[Johan Sjöberg <johan.sjoberg (at) deltamanagement.se>]

Hi.

This happened again yesterday morning. We found that core dumps had been created. Here is what gdb tells us about the core dumps. Does anyone have a clue of what might be causing this problem? The Xymon server is running CentOS 5.5 32-bit .

[root (at) mon01 acks]# file core.28581
core.28581: ELF 32-bit LSB core file Intel 80386, version 1 (SYSV), SVR4-style, from 'bbgen'

[root (at) mon01 acks]# gdb /usr/local/xymon/server/bin/bbgen core.28581
Reading symbols from /usr/local/xymon/server/bin/bbgen...done.

warning: .dynamic section for "/lib/libc.so.6" is not at the expected address

warning: difference appears to be caused by prelink, adjusting expectations
Reading symbols from /lib/libpcre.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libpcre.so.0
Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/librt.so.1
Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/ld-linux.so.2
Core was generated by `bbgen --recentgifs --subpagecolumns=2 --report'.
Program terminated with signal 6, Aborted.
#0 0x00b2f402 in __kernel_vsyscall ()

/Johan

From: Johan Sjöberg [mailto:johan.sjoberg (at) deltamanagement.se]
Sent: den 5 oktober 2010 11:10
To: xymon (at) xymon.com
Subject: [xymon] Scheduled disable causes crash?

Hi!

During the last month, we have had some problems with Xymon when using scheduled disabled (added from the web interface).

The first problem we had was on September 17th, when hobbitd crashed while/after running the scheduled disable. We got the following error in hobbit.log
2010-09-17 05:00:00 Fatal error in select: Bad file descriptor
2010-09-17 05:00:00 Setup complete


After that incident, I enabled verbose logging for hobbitd, and turned up the logging for some other process as well. This morning at 06:00 we had a new scheduled disable. This time hobbitd just stopped logging after running the disable. The web interface did not work correctly. When clicking a test, an "Internal server error" message was displayed. Also, the bbgen test went purple (last update at 05:59:26).

The only errors I have been able to find in the logs are in bb-display.log:
2010-10-05 06:00:26 xstrdup: Cannot dup NULL string
2010-10-05 06:01:26 xstrdup: Cannot dup NULL string
2010-10-05 06:02:27 xstrdup: Cannot dup NULL string
2010-10-05 06:12:30 xstrdup: Cannot dup NULL string
2010-10-05 06:13:35 xstrdup: Cannot dup NULL string
2010-10-05 06:14:39 xstrdup: Cannot dup NULL string
2010-10-05 06:24:40 xstrdup: Cannot dup NULL string
2010-10-05 06:25:41 xstrdup: Cannot dup NULL string
2010-10-05 06:26:42 xstrdup: Cannot dup NULL string
2010-10-05 06:36:47 xstrdup: Cannot dup NULL string
2010-10-05 06:37:48 xstrdup: Cannot dup NULL string
2010-10-05 06:39:54 xstrdup: Cannot dup NULL string
2010-10-05 06:40:54 xstrdup: Cannot dup NULL string
2010-10-05 06:41:54 xstrdup: Cannot dup NULL string

Xymon was restarted at 06:43.
Here is the hobbit.log from the time. It was too large to paste in the mail.
http://pastebin.com/JuU0BHje

We are running Xymon 4.2.3 on CentOS 5.

Best regards,
Johan Sjöberg