[Xymon] Too much data for svcstatus.sh?

[Henrik Størner]

Hi,

the best way to debug this is to get a proper stack trace of when the
program crashes. For this, you first need to make sure you have a binary
of svcstatus.cgi which includes debug information - if you look at the
Xymon sources, the build/Makefile.Linux file has a commented-out line
setting the CFLAGS variable for debugging:

# Compile flags for debugging
# CFLAGS = -g -DDEBUG -Wall -D_REENTRANT $(LFSDEF) $(OSDEF)

Uncomment the last line, run "make clean; make" and then use the
web/svcstatus.cgi binary from that.

To run the program in the gdb debugger, I would bypass the web server
CGI and run it by hand like this:

$ export SCRIPT_NAME="svcstatus.sh"
$ export REQUEST_METHOD="GET"
$ export QUERY_STRING="HOST=claudio.hswn.dk&SERVICE=disk"
$ gdb ./web/svcstatus.cgi
(gdb) run

The value for the QUERY_STRING should be taken from the URL for your
status-page - the part after the '?' sign. The example here is from the
xymon.com site, but you really just need to modify the hostname to that
of your own.

Note that you must run it on the Xymon server, because it reads the file
stored there.


Regards,
Henrik

Den 21.05.2020 kl. 17.47 skrev Mike Burger:
> Good morning.
>
> Running Xymon 4.3.30, compiled from source tarball, on a RHEL 6.10
> server.
>
> This morning, I encountered an interesting issue.
>
> I have a monitored system that has 197 discrete filesystems/mounts.
>
> When I attempt to enter this server's "Trends" section, I receive an
> internal server error (500).
>
> My httpd error log shows this when I attempt to access the trends link
> for the server in question:
>
> [Thu May 21 11:29:41 2020] [error] [client 10.40.24.55] Premature end
> of script headers: svcstatus.sh, referer:
> http://sysmon.fhmc.local/xymon/edmprod/edmprod.html
> [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] *** glibc
> detected *** /apps/xymon/server/bin/svcstatus.cgi: realloc(): invalid
> next size: 0x0000000000de7dd0 ***, referer:
> http://sysmon.fhmc.local/xymon/edmprod/edmprod.html
> [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55]
> /apps/xymon/server/bin/svcstatus.cgi[0x408a4f], referer:
> http://sysmon.fhmc.local/xymon/edmprod/edmprod.html
> [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55]
> /apps/xymon/server/bin/svcstatus.cgi[0x403453], referer:
> http://sysmon.fhmc.local/xymon/edmprod/edmprod.html
> [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55]
> /apps/xymon/server/bin/svcstatus.cgi[0x404bb7], referer:
> http://sysmon.fhmc.local/xymon/edmprod/edmprod.html
> [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55]
> /apps/xymon/server/bin/svcstatus.cgi[0x4022c9], referer:
> http://sysmon.fhmc.local/xymon/edmprod/edmprod.html
> [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55]
> 00400000-0043c000 r-xp 00000000 fd:08
> 24906                             
> /apps/xymon/server/bin/svcstatus.cgi, referer:
> http://sysmon.fhmc.local/xymon/edmprod/edmprod.html
> [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55]
> 0063c000-0063e000 rw-p 0003c000 fd:08
> 24906                             
> /apps/xymon/server/bin/svcstatus.cgi, referer:
> http://sysmon.fhmc.local/xymon/edmprod/edmprod.html
>
> This does not happen for any other monitored system in my environment.
>
> This system is one of the more critical servers in my environment, so
> I'd like to figure out what's going on and remediate it if possible.
>