<div dir="ltr"><div>Hi all,</div><div><br></div><div>Every day since we updated our
Xymon server to 4.3.29 (from 4.3.28), I've gotten an e-mail alert due to
xymond turning red that reads:</div><div><br></div><div style="margin-left:40px">red xymongen program crashed<br>
<br>
Fatal signal caught!</div><div><br></div><div>The strange thing is that
this has happened at 1:04 AM every day...like clockwork. I have xymongen
set to run every 1 minute and it has no problems running any other time
of the day. We are using the Terabithia RPMs and the Xymon server is
running RHEL 7.<br></div><div><br></div><div>I've scoured the system to
find anything that is set to run at/around that time via cron, etc. and
haven't found anything. The system logs don't show anything is happening
around that time either.</div><div><br></div><div>I turned on debug
logging for xymond and xymongen and haven't been able to find anything
unusual in either logs around that time. But it is dumping core files for xymongen every time it crashes.</div><div><br></div><div>I
used gdb to get the backtrace on all of the core files (so far) and
I've found that they all show the same thing. It shows the same host in
the backtrace too (although I'm farily confident it isn't specific or
isolated to that host but just the first one it runs into that it has
issues with when processing).</div><div><br></div><div>I've included an example
gdb output below (the most recent one) [1].<br></div><div><br></div><div>Is anyone else running into this by chance? Or any idea what might be the cause?<br></div><div><br></div><div>Thanks!</div><div><br></div><div><br></div><div>[1]</div><div># gdb -q /usr/libexec/xymon/xymongen core.16327<br>Reading symbols from /usr/libexec/xymon/xymongen...Reading symbols from /usr/lib/debug/usr/libexec/xymon/xymongen.debug...done.<br>done.<br>[New LWP 16327]<br>[Thread debugging using libthread_db enabled]<br>Using host libthread_db library "/lib64/libthread_db.so.1".<br>Core was generated by `/usr/libexec/xymon/xymongen --reportopts=1566187200:1566273599:0:nongr --recent'.<br>Program terminated with signal 6, Aborted.<br>#0 0x00007f4657c49377 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55<br>55 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);<br>(gdb) bt<br>#0 0x00007f4657c49377 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55<br>#1 0x00007f4657c4aa68 in __GI_abort () at abort.c:90<br>#2 0x00005589375dd455 in sigsegv_handler (signum=<optimized out>) at sig.c:57<br>#3 <signal handler called><br>#4 strchrnul () at ../sysdeps/x86_64/strchrnul.S:33<br>#5 0x00007f4657c5b681 in __find_specmb (format=0xfce <Address 0xfce out of bounds>) at printf-parse.h:109<br>#6 _IO_vfprintf_internal (s=s@entry=0x7ffd5dabcc00, <br> format=format@entry=0xfce <Address 0xfce out of bounds>, ap=ap@entry=0x7ffd5dabcd38) at vfprintf.c:1308<br>#7 0x00007f4657d28c78 in ___vsprintf_chk (s=0x7ffd5dabcf82 "", flags=1, slen=18446744073709551615, <br> format=0xfce <Address 0xfce out of bounds>, args=args@entry=0x7ffd5dabcd38) at vsprintf_chk.c:83<br>#8 0x00007f4657d28bcd in ___sprintf_chk (s=<optimized out>, flags=flags@entry=1, <br> slen=slen@entry=18446744073709551615, format=<optimized out>) at sprintf_chk.c:32<br>#9 0x00005589375ce8ca in sprintf (__fmt=<optimized out>, __s=<optimized out>)<br> at /usr/include/bits/stdio2.h:33<br>#10 parse_histlogfile (starttime=1566187200, <br> timespec=0x558937840f50 <timespec.7157> "Wed_Sep_2_19:34:55_2015", servicename=0x5589383b6d70 "procs", <br> hostname=0x558938a335d0 "<client hostname>") at availability.c:174<br>#11 parse_historyfile (fd=fd@entry=0x558938a3aea0, repinfo=<optimized out>, <br> hostname=0x558938a335d0 "<client hostname>", servicename=0x5589383b6d70 "procs", <br> fromtime=<optimized out>, totime=1566273599, for_history=for_history@entry=0, warnlevel=97, <br> greenlevel=99.995000000000005, warnstops=-1, reporttime=0x0) at availability.c:475<br>#12 0x00005589375c38cc in init_state (filename=<optimized out>, <br> filename@entry=0x7ffd5dacf210 "<client hostname>.procs", log=log@entry=0x7ffd5dacf120)<br> at loaddata.c:275<br>#13 0x00005589375c45ee in load_state (sumhead=sumhead@entry=0x558937809d48 <dispsums>) at loaddata.c:626<br>#14 0x00005589375be6f4 in main (argc=5, argv=0x7ffd5dad4418) at xymongen.c:599</div><div><br></div><div><br></div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div>-- <br></div><div>Matt Vander Werf</div></div></div></div>