[Xymon] Too much data for svcstatus.sh?

Thu May 28 16:58:47 CEST 2020

On 2020-05-22 07:16, Mark O. Stitson wrote:
> I have seen similar behaviour last week on one of our Solaris servers 
> with
> many mounts where trends fails.
> 
> Interestingly we run 2 Xymon servers for redundancy and it works 
> flawlessly
> on the secondary with exactly the same data.
> 
> When I run scvstatus.sh manually with strace a working machine all the 
> RRDs
> and then continues, the broken monitored machine throws a segfault:
> stat("./zfs.......rrd", {st_mode=S_IFREG|0644, st_size=152528, ...}) = 
> 0
> getdents(3, /* 0 entries */, 32768)     = 0
> close(3)                                = 0
> brk(0x282d000)                          = 0x282d000
> brk(0x2828000)                          = 0x2828000
> brk(0x2849000)                          = 0x2849000
> open("/usr/lib/xymon/server/web/trends_header", O_RDONLY) = 3
> fstat(3, {st_mode=S_IFREG|0644, st_size=1276, ...}) = 0
> read(3, "<!DOCTYPE HTML PUBLIC \"-//W3C//D"..., 1276) = 1276
> close(3)
> ...
> 
> Whereas the broken one segfaults:
> stat("./zfs.........rrd", {st_mode=S_IFREG|0644, st_size=76592, ...}) = 
> 0
> getdents(3, /* 0 entries */, 32768)     = 0
> close(3)                                = 0
> brk(0x266b000)                          = 0x266b000
> brk(0x2666000)                          = 0x2666000
> mmap(NULL, 159744, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 
> 0) =
> 0x7f82e2f29000
> brk(0x2661000)                          = 0x2661000
> brk(0x265f000)                          = 0x265f000
> --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x26631f8} 
> ---
> +++ killed by SIGSEGV +++
> 
> It is on my list to investigate, but I haven't had time yet.
> 
> Mark
> 
> -----Original Message-----
> From: Xymon <xymon-bounces at xymon.com> On Behalf Of Mike Burger
> Sent: 21 May 2020 16:47
> To: xymon at xymon.com
> Subject: [Xymon] Too much data for svcstatus.sh?
> 
> Good morning.
> 
> Running Xymon 4.3.30, compiled from source tarball, on a RHEL 6.10 
> server.
> 
> This morning, I encountered an interesting issue.
> 
> I have a monitored system that has 197 discrete filesystems/mounts.
> 
> When I attempt to enter this server's "Trends" section, I receive an
> internal server error (500).
> 
> My httpd error log shows this when I attempt to access the trends link 
> for
> the server in question:
> 
> [Thu May 21 11:29:41 2020] [error] [client 10.40.24.55] Premature end 
> of
> script headers: svcstatus.sh, referer:
> http://sysmon.fhmc.local/xymon/edmprod/edmprod.html
> [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] *** glibc 
> detected
> *** /apps/xymon/server/bin/svcstatus.cgi: realloc(): invalid next size:
> 0x0000000000de7dd0 ***, referer:
> http://sysmon.fhmc.local/xymon/edmprod/edmprod.html
> [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55]
> /apps/xymon/server/bin/svcstatus.cgi[0x408a4f], referer:
> http://sysmon.fhmc.local/xymon/edmprod/edmprod.html
> [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55]
> /apps/xymon/server/bin/svcstatus.cgi[0x403453], referer:
> http://sysmon.fhmc.local/xymon/edmprod/edmprod.html
> [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55]
> /apps/xymon/server/bin/svcstatus.cgi[0x404bb7], referer:
> http://sysmon.fhmc.local/xymon/edmprod/edmprod.html
> [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55]
> /apps/xymon/server/bin/svcstatus.cgi[0x4022c9], referer:
> http://sysmon.fhmc.local/xymon/edmprod/edmprod.html
> [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55]
> 00400000-0043c000 r-xp 00000000 fd:08 24906
> /apps/xymon/server/bin/svcstatus.cgi, referer:
> http://sysmon.fhmc.local/xymon/edmprod/edmprod.html
> [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55]
> 0063c000-0063e000 rw-p 0003c000 fd:08 24906
> /apps/xymon/server/bin/svcstatus.cgi, referer:
> http://sysmon.fhmc.local/xymon/edmprod/edmprod.html
> 
> This does not happen for any other monitored system in my environment.
> 
> This system is one of the more critical servers in my environment, so 
> I'd
> like to figure out what's going on and remediate it if possible.

Henrik, et al:

Any thoughts?
-- 
Mike Burger
http://www.bubbanfriends.org

"It's always suicide-mission this, save-the-planet that. No one ever 
just stops by to say 'hi' anymore." --Colonel Jack O'Neill, SG1