<div dir="auto"><div>How is the data from the Synology device being processed into RRD? Are you using NCV or the "--extra-script" method?</div><div dir="auto"><br></div><div dir="auto">I ran into a similar RRD crash when upgrading my TST Xymon environment from 4.3.28 to both 4.3.29 and .30. I tracked it down to a bug with my extra-script for a custom test. I reworked the custom test to use NCV and the crash stopped. </div><div dir="auto"><br><div class="gmail_quote" dir="auto"><div dir="ltr" class="gmail_attr">On Thu, Oct 17, 2019, 7:38 AM Andrey Chervonets <<a href="mailto:A.Chervonets@cominder.eu">A.Chervonets@cominder.eu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><font size="2" face="sans-serif">To get more information I have enabled
"--debug" to both channels (status and data).</font>
<br><font size="2" face="sans-serif">Then we see a bit more information in
rrd-status.log:</font>
<br><font size="2" face="sans-serif">....</font>
<br><font size="2" face="sans-serif">2019-10-17 13:40:02.376153 Host '<a href="http://synologyhost.domain.eu" target="_blank" rel="noreferrer">synologyhost.domain.eu</a>'
reports netstat for an unknown OS</font>
<br><font size="2" face="sans-serif">408 2019-10-17 13:40:02.376181 Flush,
but xymonmsg is empty</font>
<br><font size="2" face="sans-serif">408 2019-10-17 13:40:02.376185 0 status
messages merged into 1 transmissions</font>
<br><font size="2" face="sans-serif">408 2019-10-17 13:40:02.376203 xymond_rrd:
Got message 612 @@status#612/<a href="http://synologyhost.domain.eu" target="_blank" rel="noreferrer">synologyhost.domain.eu</a>|1571308802.357389|83.99.221.6||<a href="http://synologyhost.domain.eu" target="_blank" rel="noreferrer">synologyhost.domain.eu</a>|procs|1571326802|green||green|1570620002|0||0||1571051696||p_cominder|0|</font>
<br><font size="2" face="sans-serif">408 2019-10-17 13:40:02.376210 startpos
95710, fillpos 99309, endpos 97006</font>
<br><font size="2" face="sans-serif">408 2019-10-17 13:40:02.376227 Flush,
but xymonmsg is empty</font>
<br><font size="2" face="sans-serif">408 2019-10-17 13:40:02.376233 0 status
messages merged into 1 transmissions</font>
<br><font size="2" face="sans-serif">408 2019-10-17 13:40:02.376244 xymond_rrd:
Got message 613 @@status#613/<a href="http://synologyhost.domain.eu" target="_blank" rel="noreferrer">synologyhost.domain.eu</a>|1571308802.357673|83.99.221.6||<a href="http://synologyhost.domain.eu" target="_blank" rel="noreferrer">synologyhost.domain.eu</a>|raid|1571326802|green||green|1570620002|0||0||1571051696||p_cominder|0|</font>
<br><font size="2" face="sans-serif">408 2019-10-17 13:40:02.376251 startpos
97010, fillpos 99309, endpos 97945</font>
<br><font size="2" face="sans-serif">408 2019-10-17 13:40:02.376269 Flush,
but xymonmsg is empty</font>
<br><font size="2" face="sans-serif">408 2019-10-17 13:40:02.376276 0 status
messages merged into 1 transmissions</font>
<br><font size="2" face="sans-serif">408 2019-10-17 13:40:02.376288 xymond_rrd:
Got message 614 @@status#614/<a href="http://synologyhost.domain.eu" target="_blank" rel="noreferrer">synologyhost.domain.eu</a>|1571308802.368308|83.99.221.6||<a href="http://synologyhost.domain.eu" target="_blank" rel="noreferrer">synologyhost.domain.eu</a>|temperature|1571326802|green||green|1570620002|0||0||1571051696||p_cominder|0|</font>
<br><font size="2" face="sans-serif">408 2019-10-17 13:40:02.376294 startpos
97949, fillpos 99309, endpos 98645</font>
<br><font size="2" face="sans-serif">2019-10-17 13:40:02.381339 Child process
408 died: Signal 6</font>
<br><font size="2" face="sans-serif">2019-10-17 13:40:04.432302 Peer at <a href="http://0.0.0.0:0" target="_blank" rel="noreferrer">0.0.0.0:0</a>
failed: Broken pipe</font>
<br><font size="2" face="sans-serif">2019-10-17 13:40:04.452708 Peer not
up, flushing message queue</font>
<br><font size="2" face="sans-serif">13920 2019-10-17 13:40:04.557656 setup_feedback_queue:
got ID -1 for key 0xA03EB91</font>
<br><font size="2" face="sans-serif">13920 2019-10-17 13:40:04.558141 Opening
file /u01/app/xymon/product/xymon4.3.30/server/etc/rrddefinitions.cfg</font>
<br><font size="2" face="sans-serif">13920 2019-10-17 13:40:04.558326 Want
msg 1, startpos 0, fillpos 0, endpos -1, usedbytes=0, bufleft=1052671</font>
<br><font size="2" face="sans-serif">13920 2019-10-17 13:40:04.558359 Got
6716 bytes</font>
<br><font size="2" face="sans-serif">...</font>
<br><font size="2" face="sans-serif">Here we can see processing of data from
our Synology NAS with Synology Monitoring Tool 1.4.8, </font><a href="http://www.sysco.ch/synomon/" target="_blank" rel="noreferrer"><font size="2" color="blue" face="sans-serif">http://www.sysco.ch/synomon/</font></a><font size="2" face="sans-serif">
enabled.</font>
<br><font size="2" face="sans-serif">Make note - despite RRD crash we can
see good status and text of "temperature" metric status like:</font>
<br><font size="2" face="sans-serif">--</font>
<br><font size="2" face="sans-serif">Device
Temp(C) Temp(F)</font>
<br><font size="2" face="sans-serif">---------------------------------------</font>
<br><font size="2" face="sans-serif">green system
52 125</font>
<br><font size="2" face="sans-serif">green /dev/sda
36 96</font>
<br><font size="2" face="sans-serif">green /dev/sdb
38 100</font>
<br><font size="2" face="sans-serif">green /dev/sdd
36 96</font>
<br><font size="2" face="sans-serif">---------------------------------------</font>
<br>
<br><font size="2" face="sans-serif">Synology Monitoring Tool 1.4.8, </font><a href="http://www.sysco.ch/synomon/" target="_blank" rel="noreferrer"><font size="2" face="sans-serif">http://www.sysco.ch/synomon/</font></a>
<br><font size="2" face="sans-serif">Model: RS812+ (synologyhost,<a href="http://domain.eu" target="_blank" rel="noreferrer">domain.eu</a>)</font>
<br><font size="2" face="sans-serif">Processor: Intel(R) Atom(TM) CPU D2701
@ 2.13GHz</font>
<br><font size="2" face="sans-serif">System temperature: 52°C</font>
<br><font size="2" face="sans-serif">Serial number: serialnumberdata-replaced</font>
<br><font size="2" face="sans-serif">Firmware: 6.2-24922</font>
<br><font size="2" face="sans-serif">MAC address(s): number-replaced, number-replaced</font>
<br><font size="2" face="sans-serif">Linux version 3.10.105 (root@build10)
(gcc version 4.9.3 20150311 (prerelease) (crosstool-NG 1.20.0) ) #24922
SMP Fri May 10 02:51:01 CST 2019</font>
<br><font size="2" face="sans-serif">--</font>
<br>
<br><font size="2" face="sans-serif">After stopping the plugin on Synology
we have got no more data from it and no more xymond_rrd crash (red changed
to purple, as expected).</font>
<br>
<br><font size="2" face="sans-serif">I am note sure where is the problem/bug.
So I have added the Synology Monitoring Tool developers e-mail to our communictaion.</font>
<br>
<br><font size="2" face="sans-serif">Please, review and give the hint how
can we fix the problem - our NAS state monitoring is quite critical
thing we need.</font>
<br>
<br><font size="2" face="sans-serif">The suspection has been also proved
by GDC info (as instructed at: </font><a href="http://www.robertandrobert.com/xymon/help/known-issues.html" target="_blank" rel="noreferrer"><font size="2" color="blue" face="sans-serif">http://www.robertandrobert.com/xymon/help/known-issues.html</font></a><font size="2" face="sans-serif">
):</font>
<br><font size="2" face="sans-serif">--</font>
<br><font size="2" face="sans-serif">[xymon@synologyhost server]$ /bin/gdb
/u01/app/xymon/product/xymon4.3.30/server/bin/xymond_rrd tmp/core.408</font>
<br><font size="2" face="sans-serif">GNU gdb (GDB) Red Hat Enterprise Linux
7.6.1-115.el7</font>
<br><font size="2" face="sans-serif">... copyright...</font>
<br><font size="2" face="sans-serif">...</font>
<br><font size="2" face="sans-serif">Reading symbols from /u01/app/xymon/product/xymon4.3.30/server/bin/xymond_rrd...done.</font>
<br><font size="2" face="sans-serif">[New LWP 408]</font>
<br><font size="2" face="sans-serif">[Thread debugging using libthread_db
enabled]</font>
<br><font size="2" face="sans-serif">Using host libthread_db library "/lib64/libthread_db.so.1".</font>
<br><font size="2" face="sans-serif">Core was generated by `xymond_rrd --rrddir=/u01/app/xymon/product/xymon4.3.30/data/rrd
--debug'.</font>
<br><font size="2" face="sans-serif">Program terminated with signal 6, Aborted.</font>
<br><font size="2" face="sans-serif">#0 0x00007f62fcd85337 in raise
() from /lib64/libc.so.6</font>
<br><font size="2" face="sans-serif">Missing separate debuginfos, use: debuginfo-install
bzip2-libs-1.0.6-13.el7.x86_64 cairo-1.15.12-4.el7.x86_64 expat-2.1.0-10.el7_3.x86_64
fontconfig-2.13.0-4.3.el7.x86_64 freetype-2.8-14.el7.x86_64 fribidi-1.0.2-1.el7.x86_64
glib2-2.56.1-5.el7.x86_64 glibc-2.17-292.el7.x86_64 graphite2-1.3.10-1.el7_3.x86_64
harfbuzz-1.7.5-2.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-37.el7_7.2.x86_64
libX11-1.6.7-2.el7.x86_64 libXau-1.0.8-2.1.el7.x86_64 libXext-1.3.3-3.el7.x86_64
libXrender-0.9.10-1.el7.x86_64 libcom_err-1.42.9-16.el7.x86_64 libffi-3.0.13-18.el7.x86_64
libgcc-4.8.5-39.el7.x86_64 libglvnd-1.0.1-0.8.git5baa1e5.el7.x86_64 libglvnd-egl-1.0.1-0.8.git5baa1e5.el7.x86_64
libglvnd-glx-1.0.1-0.8.git5baa1e5.el7.x86_64 libpng-1.5.13-7.el7_2.x86_64
libselinux-2.5-14.1.el7.x86_64 libthai-0.1.14-9.el7.x86_64 libtirpc-0.2.4-0.16.el7.x86_64
libuuid-2.23.2-61.el7.x86_64 libxcb-1.13-1.el7.x86_64 libxml2-2.9.1-6.el7_2.3.x86_64
openssl-libs-1.0.2k-19.el7.x86_64 pango-1.42.4-4.el7_7.x86_64 pcre-8.32-17.el7.x86_64
pixman-0.34.0-1.el7.x86_64 rrdtool-1.4.8-9.el7.x86_64 xz-libs-5.2.2-1.el7.x86_64
zlib-1.2.7-18.el7.x86_64</font>
<br><font size="2" face="sans-serif">(gdb)</font>
<br><font size="2" face="sans-serif">(gdb)</font>
<br><font size="2" face="sans-serif">(gdb) bt</font>
<br><font size="2" face="sans-serif">#0 0x00007f62fcd85337 in raise
() at /lib64/libc.so.6</font>
<br><font size="2" face="sans-serif">#1 0x00007f62fcd86a28 in abort
() at /lib64/libc.so.6</font>
<br><font size="2" face="sans-serif">#2 0x0000000000428e63 in sigsegv_handler
(signum=<optimized out>) at sig.c:57</font>
<br><font size="2" face="sans-serif">#3 0x00007f62fcd853b0 in <signal
handler called> () at /lib64/libc.so.6</font>
<br><font size="2" face="sans-serif">#4 0x00007f62fcd89f97 in ____strtoll_l_internal
() at /lib64/libc.so.6</font>
<br><font size="2" face="sans-serif">#5 0x000000000040f9c2 in do_temperature_rrd
(__nptr=0x0) at /usr/include/stdlib.h:280</font>
<br><font size="2" face="sans-serif">#6 0x000000000040f9c2 in do_temperature_rrd
(hostname=hostname@entry=0x7f62fdfceb43 "<a href="http://synologyhost.domain.eu" target="_blank" rel="noreferrer">synologyhost.domain.eu</a>",
testname=testname@entry=0x7f62fdfceb58 "temperature", classname=classname@entry=0x7f62fdfceb99
"p_cominder", pagepaths=pagepaths@entry=0x7f62fdfceba4 "0",
msg=msg@entry=0x7f62fdfceba7 "status+300 synologyhost,domain.eu.temperature
green 2019-10-17 13:40:01 [<a href="http://synologyhost.domain.eu" target="_blank" rel="noreferrer">synologyhost.domain.eu</a>] - temperature\nDevice",
' ' <repeats 13 times>, "Temp(C) Temp(F)\n", '-'
<repeats 39 times>, "\n&green system"...,
tstamp=tstamp@entry=1571308802) at rrd/do_temperature.c:100</font>
<br><font size="2" face="sans-serif">#7 0x000000000041316b in update_rrd
(hostname=hostname@entry=0x7f62fdfceb43 "<a href="http://synologyhost.domain.eu" target="_blank" rel="noreferrer">synologyhost.domain.eu</a>",
testname=<optimized out>,</font>
<br><font size="2" face="sans-serif"> testname@entry=0x7f62fdfceb58
"temperature", msg=msg@entry=0x7f62fdfceba7 "status+300
synologyhost,domain.eu.temperature green 2019-10-17 13:40:01 [<a href="http://synologyhost.domain.eu" target="_blank" rel="noreferrer">synologyhost.domain.eu</a>]
- temperature\nDevice", ' ' <repeats 13 times>, "Temp(C)
Temp(F)\n", '-' <repeats 39 times>, "\n&green
system"..., tstamp=tstamp@entry=1571308802, sender=sender@entry=0x7f62fdfceb36
"83.99.221.6", ldef=<optimized out>, classname=classname@entry=0x7f62fdfceb99
"p_cominder", pagepaths=pagepaths@entry=0x7f62fdfceba4 "0")
at do_rrd.c:714</font>
<br><font size="2" face="sans-serif">#8 0x0000000000403434 in main
(argc=<optimized out>, argv=0x7ffffb4bd4b8) at xymond_rrd.c:391</font>
<br><font size="2" face="sans-serif">(gdb)</font>
<br><font size="2" face="sans-serif">--</font>
<br>
<br><font size="2" face="Verdana">So, we know which metric cause RRD crash,
we have workaround (to make RRD working to generate other metrics graphs),</font>
<br><font size="2" face="Verdana">but we need better solution to make all
that working as expected.</font>
<br>
<br><font size="2" face="Verdana">P.S. Note: real hostname is replaced in
all outputs submitted in e-mail (just if there are some checksums are used).</font>
<br>
<br>
<br><font size="2" face="Verdana">Best regards,</font>
<br>
<br><font size="2" face="Verdana">Andrey Chervonets</font>
<br><font size="2" face="Verdana">----------------------</font>
<br><font size="2" face="Verdana">CoMinder Support</font>
<br><a href="http://www.cominder.eu/" target="_blank" rel="noreferrer"><font size="2" color="blue" face="Verdana">http://www.cominder.eu/</font></a>
<br><font size="2" face="Verdana">mobile: +371 26517848</font>
<br>
<br><font size="2" face="sans-serif"> </font>
<br>
<br>
<br><tt><font size="2">"Xymon" <<a href="mailto:xymon-bounces@xymon.com" target="_blank" rel="noreferrer">xymon-bounces@xymon.com</a>>
wrote on 15.10.2019 13:00:01:<br>
<br>
> From: <a href="mailto:xymon-request@xymon.com" target="_blank" rel="noreferrer">xymon-request@xymon.com</a></font></tt>
<br><tt><font size="2">> To: <a href="mailto:xymon@xymon.com" target="_blank" rel="noreferrer">xymon@xymon.com</a></font></tt>
<br><tt><font size="2">> Date: 15.10.2019 13:00</font></tt>
<br><tt><font size="2">> Subject: Xymon Digest, Vol 105, Issue 9</font></tt>
<br><tt><font size="2">> Sent by: "Xymon" <<a href="mailto:xymon-bounces@xymon.com" target="_blank" rel="noreferrer">xymon-bounces@xymon.com</a>></font></tt>
<br><tt><font size="2">> <br>
> ----------------------------------------------------------------------<br>
> <br>
> Message: 1<br>
> Date: Mon, 14 Oct 2019 15:09:53 +0300<br>
> From: Andrey Chervonets <<a href="mailto:A.Chervonets@cominder.eu" target="_blank" rel="noreferrer">A.Chervonets@cominder.eu</a>><br>
> To: <a href="mailto:xymon@xymon.com" target="_blank" rel="noreferrer">xymon@xymon.com</a><br>
> Subject: [Xymon] xymond_rrd - Program crashed after fresh install
of<br>
> Xymon 4.3.30 and data from Xymon 4.3.17<br>
> Message-ID:<br>
> <<a href="mailto:OFD5D1CD2D.3E1D4B14-ONC2258493.00408D6C-C2258493.0042D300@cominder.eu" target="_blank" rel="noreferrer">OFD5D1CD2D.3E1D4B14-ONC2258493.00408D6C-C2258493.0042D300@cominder.eu</a>><br>
> <br>
> Content-Type: text/plain; charset="us-ascii"<br>
> <br>
> Good day!<br>
> <br>
> Recently we have installed Xymon 4.3.30 on new VM (CentOS Linux release
<br>
> 7.7.1908 (Core) - guest under KVM<br>
> Guest Kernel: 3.10.0-1062.1.1.el7.x86_64 #1 SMP Fri Sep 13
22:55:44 UTC <br>
> 2019 x86_64 x86_64 x86_64 GNU/Linux<br>
> <br>
> All OK, except xymond_rrd is crashing frequently - the "xymond_rrd"
metric <br>
> is always red (was never green) with message:<br>
> - Program crashed<br>
> Fatal signal caught!<br>
> <br>
> In rrd-status.log we can find frequent messages like:<br>
> <br>
> 2019-10-14 14:35:03.609265 Child process 2997 died: Signal 6<br>
> 2019-10-14 14:35:04.239677 Peer at <a href="http://0.0.0.0:0" target="_blank" rel="noreferrer">0.0.0.0:0</a> failed: Broken pipe<br>
> 2019-10-14 14:35:08.886124 Peer not up, flushing message queue<br>
> 2019-10-14 14:36:45.883398 Host '<a href="http://synologyhost.domain.eu" target="_blank" rel="noreferrer">synologyhost.domain.eu</a>' reports netstat
<br>
> for an unknown OS<br>
> 2019-10-14 14:36:45.888875 Child process 21622 died: Signal 6<br>
> 2019-10-14 14:36:52.510319 Peer at <a href="http://0.0.0.0:0" target="_blank" rel="noreferrer">0.0.0.0:0</a> failed: Broken pipe<br>
> 2019-10-14 14:36:52.510720 Peer not up, flushing message queue<br>
> 2019-10-14 14:40:02.689062 Host '<a href="http://synologyhost.domain.eu" target="_blank" rel="noreferrer">synologyhost.domain.eu</a>' reports netstat
<br>
> for an unknown OS<br>
> 2019-10-14 14:40:02.694320 Child process 28158 died: Signal 6<br>
> 2019-10-14 14:40:05.119354 Peer at <a href="http://0.0.0.0:0" target="_blank" rel="noreferrer">0.0.0.0:0</a> failed: Broken pipe<br>
> 2019-10-14 14:40:05.250422 Peer not up, flushing message queue<br>
> <br>
> Note: lines like "Host '<a href="http://synologyhost.domain.eu" target="_blank" rel="noreferrer">synologyhost.domain.eu</a>' reports netstat
for an <br>
> unknown OS" are comining from Synonlogy NAS with Monitoring package
<br>
> installed.<br>
> I am sure it is not related - it was working on old Xymon 4.3.17 (CentOS
<br>
> 6.6)<br>
> <br>
> After fresh installation we just remapped (with symbolic link) the
data <br>
> directory to continue employ old data logs and rra.<br>
> <br>
> There is plenty of core files under server/tmp/<br>
> srw-rw-rw- 1 xymon monitor 0 Oct 14 14:40 rrdctl.572<br>
> -rw------- 1 xymon monitor 3252224 Oct 14 14:45 core.572<br>
> srw-rw-rw- 1 xymon monitor 0 Oct 14 14:45 rrdctl.17027<br>
> -rw------- 1 xymon monitor 3248128 Oct 14 14:50 core.17027<br>
> srw-rw-rw- 1 xymon monitor 0 Oct 14 14:50 rrdctl.30574<br>
> -rw------- 1 xymon monitor 3248128 Oct 14 14:55 core.30574<br>
> srw-rw-rw- 1 xymon monitor 0 Oct 14 14:55 rrdctl.13275<br>
> -rw------- 1 xymon monitor 3239936 Oct 14 15:00 core.13275<br>
> -rw-r--r-- 1 xymon monitor 1887355 Oct 14 15:02 xymond.chk<br>
> -rw-r--r-- 1 xymon monitor 0 Oct 14 15:02 alert.chk.sub<br>
> -rw-r--r-- 1 xymon monitor 70921 Oct 14 15:02 alert.chk<br>
> srw-rw-rw- 1 xymon monitor 0 Oct 14 15:02 rrdctl.5887<br>
> srw-rw-rw- 1 xymon monitor 0 Oct 14 15:02 rrdctl.5954<br>
> -rw------- 1 xymon monitor 3764224 Oct 14 15:05 core.5887<br>
> srw-rw-rw- 1 xymon monitor 0 Oct 14 15:05 rrdctl.10234<br>
> <br>
> <br>
> Question: How can we diagnose what is the cause of the problem?<br>
> <br>
> <br>
> <br>
> Best regards,<br>
> <br>
> Andrey Chervonets<br>
> ----------------------<br>
> SIA CoMinder<br>
> </font></tt><a href="http://www.cominder.eu/" target="_blank" rel="noreferrer"><tt><font size="2">http://www.cominder.eu/</font></tt></a><tt><font size="2"><br>
> mobile: +371 26517848<br>
> -------------- next part --------------<br>
</font></tt>
_______________________________________________<br>
Xymon mailing list<br>
<a href="mailto:Xymon@xymon.com" target="_blank" rel="noreferrer">Xymon@xymon.com</a><br>
<a href="http://lists.xymon.com/mailman/listinfo/xymon" rel="noreferrer noreferrer" target="_blank">http://lists.xymon.com/mailman/listinfo/xymon</a><br>
</blockquote></div></div></div>