[Xymon] Too much data for svcstatus.sh?

Fri May 22 13:16:44 CEST 2020

I have seen similar behaviour last week on one of our Solaris servers with
many mounts where trends fails.

Interestingly we run 2 Xymon servers for redundancy and it works flawlessly
on the secondary with exactly the same data.

When I run scvstatus.sh manually with strace a working machine all the RRDs
and then continues, the broken monitored machine throws a segfault:
stat("./zfs.......rrd", {st_mode=S_IFREG|0644, st_size=152528, ...}) = 0
getdents(3, /* 0 entries */, 32768)     = 0
close(3)                                = 0
brk(0x282d000)                          = 0x282d000
brk(0x2828000)                          = 0x2828000
brk(0x2849000)                          = 0x2849000
open("/usr/lib/xymon/server/web/trends_header", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=1276, ...}) = 0
read(3, "<!DOCTYPE HTML PUBLIC \"-//W3C//D"..., 1276) = 1276
close(3)  
...

Whereas the broken one segfaults:
stat("./zfs.........rrd", {st_mode=S_IFREG|0644, st_size=76592, ...}) = 0
getdents(3, /* 0 entries */, 32768)     = 0
close(3)                                = 0
brk(0x266b000)                          = 0x266b000
brk(0x2666000)                          = 0x2666000
mmap(NULL, 159744, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7f82e2f29000
brk(0x2661000)                          = 0x2661000
brk(0x265f000)                          = 0x265f000
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x26631f8} ---
+++ killed by SIGSEGV +++

It is on my list to investigate, but I haven't had time yet.

Mark

-----Original Message-----
From: Xymon <xymon-bounces at xymon.com> On Behalf Of Mike Burger
Sent: 21 May 2020 16:47
To: xymon at xymon.com
Subject: [Xymon] Too much data for svcstatus.sh?

Good morning.

Running Xymon 4.3.30, compiled from source tarball, on a RHEL 6.10 server.

This morning, I encountered an interesting issue.

I have a monitored system that has 197 discrete filesystems/mounts.

When I attempt to enter this server's "Trends" section, I receive an
internal server error (500).

My httpd error log shows this when I attempt to access the trends link for
the server in question:

[Thu May 21 11:29:41 2020] [error] [client 10.40.24.55] Premature end of
script headers: svcstatus.sh, referer: 
http://sysmon.fhmc.local/xymon/edmprod/edmprod.html
[Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] *** glibc detected
*** /apps/xymon/server/bin/svcstatus.cgi: realloc(): invalid next size:
0x0000000000de7dd0 ***, referer: 
http://sysmon.fhmc.local/xymon/edmprod/edmprod.html
[Thu May 21 11:29:59 2020] [error] [client 10.40.24.55]
/apps/xymon/server/bin/svcstatus.cgi[0x408a4f], referer: 
http://sysmon.fhmc.local/xymon/edmprod/edmprod.html
[Thu May 21 11:29:59 2020] [error] [client 10.40.24.55]
/apps/xymon/server/bin/svcstatus.cgi[0x403453], referer: 
http://sysmon.fhmc.local/xymon/edmprod/edmprod.html
[Thu May 21 11:29:59 2020] [error] [client 10.40.24.55]
/apps/xymon/server/bin/svcstatus.cgi[0x404bb7], referer: 
http://sysmon.fhmc.local/xymon/edmprod/edmprod.html
[Thu May 21 11:29:59 2020] [error] [client 10.40.24.55]
/apps/xymon/server/bin/svcstatus.cgi[0x4022c9], referer: 
http://sysmon.fhmc.local/xymon/edmprod/edmprod.html
[Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] 
00400000-0043c000 r-xp 00000000 fd:08 24906                              
/apps/xymon/server/bin/svcstatus.cgi, referer: 
http://sysmon.fhmc.local/xymon/edmprod/edmprod.html
[Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] 
0063c000-0063e000 rw-p 0003c000 fd:08 24906                              
/apps/xymon/server/bin/svcstatus.cgi, referer: 
http://sysmon.fhmc.local/xymon/edmprod/edmprod.html

This does not happen for any other monitored system in my environment.

This system is one of the more critical servers in my environment, so I'd
like to figure out what's going on and remediate it if possible.

--
Mike Burger
http://www.bubbanfriends.org

"It's always suicide-mission this, save-the-planet that. No one ever just
stops by to say 'hi' anymore." --Colonel Jack O'Neill, SG1
_______________________________________________
Xymon mailing list
Xymon at xymon.com
http://lists.xymon.com/mailman/listinfo/xymon