<div dir="ltr"><div>Looking at the logs below, it appears something might be broken in the [telnet] definition.<br></div><div>Here's the first few lines from my protocols.cfg file<br></div><div><font size="1"><span style="font-family:monospace,monospace">[ftp]<br> send "quit\r\n"<br> expect "220"<br> options banner<br> port 21<br><br>[ftps]<br> send "quit\r\n"<br> expect "220"<br> options ssl,banner<br> port 990<br><br>[ssh|ssh1|ssh2]<br> send "SSH-2.0-OpenSSH_4.1\r\n"<br> expect "SSH"<br> options banner<br> port 22<br><br>[telnet]<br> options banner,telnet<br> port 23</span></font><br><br><br>From rrd-data.log<br><font size="1"><span style="font-family:monospace,monospace">5614 2015-03-11 08:29:07 Opening file /opt/local/xymon/server/etc/rrddefinitions.cfg<br>5614 2015-03-11 08:29:07 Want msg 1, startpos 0, fillpos 0, endpos -1, usedbytes=0, bufleft=2101247<br>5614 2015-03-11 08:29:07 Got 403 bytes<br>5614 2015-03-11 08:29:07 xymond_rrd: Got message 849930 @@data#849930/transfer|1426033746.620045|172.21.219.192||transfer|iostatdisk|sunos|unix/UnixInf<br>5614 2015-03-11 08:29:07 startpos 403, fillpos 403, endpos -1<br>5614 2015-03-11 08:29:07 Transport setup is:<br>5614 2015-03-11 08:29:07 xymondportnumber = 1984<br>5614 2015-03-11 08:29:07 xymonproxyhost = NONE<br>5614 2015-03-11 08:29:07 xymonproxyport = 0<br>5614 2015-03-11 08:29:07 Recipient listed as '172.21.219.192'<br>5614 2015-03-11 08:29:07 Standard protocol on port 1984<br>5614 2015-03-11 08:29:07 Will connect to address 172.21.219.192 port 1984<br>5614 2015-03-11 08:29:07 Connect status is 0<br>5614 2015-03-11 08:29:07 Sent 16 bytes<br>5614 2015-03-11 08:29:07 Read 8192 bytes<br>5614 2015-03-11 08:29:07 Read 7657 bytes<br>5614 2015-03-11 08:29:07 Closing connection<br>5614 2015-03-11 08:29:07 Opening file /opt/local/xymon/server/etc/analysis.cfg<br>5614 2015-03-11 08:29:07 Compiling regex ctrldom[A-D]<br>5614 2015-03-11 08:29:07 Compiling regex infdom[A-B]1<br>5614 2015-03-11 08:29:07 Compiling regex .+dom..<br>5614 2015-03-11 08:29:07 Compiling regex infdom[A-B]1|notesdomD1<br>5614 2015-03-11 08:29:07 Compiling regex [.:]80$<br>5614 2015-03-11 08:29:07 Compiling regex syn.*,%oradom.*<br>5614 2015-03-11 08:29:07 Compiling regex syn.*|oradom.*|SINFV.*<br>5614 2015-03-11 08:29:07 Compiling regex [.:]514$<br>5614 2015-03-11 08:29:07 Compiling regex [.:]3389$<br>5614 2015-03-11 08:29:07 Compiling regex energise.*<br>5614 2015-03-11 08:29:07 Compiling regex [.:]8989$<br>5614 2015-03-11 08:29:07 Compiling regex [.:]80$<br>5614 2015-03-11 08:29:07 Compiling regex [.:]4848$<br>5614 2015-03-11 08:29:07 Compiling regex [.:]8080$<br>5614 2015-03-11 08:29:07 Compiling regex ^/var/crash/.*<br>5614 2015-03-11 08:29:07 Compiling regex ^/var/core/.*<br>5614 2015-03-11 08:29:07 Compiling regex [.:]22$<br>5614 2015-03-11 08:29:07 Compiling regex sinfct.*|domino.*<br>5614 2015-03-11 08:29:07 Compiling regex [.:]80$<br>5614 2015-03-11 08:29:07 Compiling regex [.:]25$<br>5614 2015-03-11 08:29:07 Compiling regex sinfct.*|domino.*|SINFV.*|hermes|sirocco<br>5614 2015-03-11 08:29:07 Compiling regex SINFV.*<br>5614 2015-03-11 08:29:07 Compiling regex .*<br>5614 2015-03-11 08:29:07 Compiling regex ^error.*<br>5614 2015-03-11 08:29:07 Compiling regex ^error.*<br>5614 2015-03-11 08:29:07 Compiling regex .*restart/shutdown.*|.*Citrix<br>5614 2015-03-11 08:29:07 Compiling regex .*<br>5614 2015-03-11 08:29:07 Compiling regex ^warning.*<br>5614 2015-03-11 08:29:07 Compiling regex ^warning.*<br>5614 2015-03-11 08:29:07 Compiling regex .*MetaFrameEvents.*|.*restart/shutdown.*|.*Citrix-Multimedia-Flash.*|.*Printer<br>5614 2015-03-11 08:29:07 Compiling regex .*<br>5614 2015-03-11 08:29:07 Compiling regex ^failure.*<br>5614 2015-03-11 08:29:07 Compiling regex ^failure.*<br>5614 2015-03-11 08:29:07 Compiling regex ^/var/crash/.*<br>5614 2015-03-11 08:29:07 Compiling regex ^/var/core/.*<br>5614 2015-03-11 08:29:07 Compiling regex [.:]22$<br>5614 2015-03-11 08:29:07 Compiling regex ^/var/crash/.*<br>5614 2015-03-11 08:29:07 Compiling regex ^/var/core/.*<br>5614 2015-03-11 08:29:07 Compiling regex [.:]22$<br>5614 2015-03-11 08:29:07 Opening file /opt/local/xymon/server/etc/protocols.cfg<br>5614 2015-03-11 08:29:07 Service list dump<br>5614 2015-03-11 08:29:07 Name : ftp<br>5614 2015-03-11 08:29:07 Sendtext: quit\r\n<br>5614 2015-03-11 08:29:07 Sendlen : 6<br>5614 2015-03-11 08:29:07 Exp.text: 220<br>5614 2015-03-11 08:29:07 Exp.len : 3<br>5614 2015-03-11 08:29:07 Exp.ofs : 0<br>5614 2015-03-11 08:29:07 Flags : 1<br>5614 2015-03-11 08:29:07 Port : 21<br>5614 2015-03-11 08:29:07 Name : ftps<br>5614 2015-03-11 08:29:07 Sendtext: quit\r\n<br>5614 2015-03-11 08:29:07 Sendlen : 6<br>5614 2015-03-11 08:29:07 Exp.text: 220<br>5614 2015-03-11 08:29:07 Exp.len : 3<br>5614 2015-03-11 08:29:07 Exp.ofs : 0<br>5614 2015-03-11 08:29:07 Flags : 5<br>5614 2015-03-11 08:29:07 Port : 990<br>5614 2015-03-11 08:29:07 Name : ssh<br>5614 2015-03-11 08:29:07 Sendtext: SSH-2.0-OpenSSH_4.1\r\n<br>5614 2015-03-11 08:29:07 Sendlen : 21<br>5614 2015-03-11 08:29:07 Exp.text: SSH<br>5614 2015-03-11 08:29:07 Exp.len : 3<br>5614 2015-03-11 08:29:07 Exp.ofs : 0<br>5614 2015-03-11 08:29:07 Flags : 1<br>5614 2015-03-11 08:29:07 Port : 22<br>5614 2015-03-11 08:29:07 Name : ssh1<br>5614 2015-03-11 08:29:07 Sendtext: SSH-2.0-OpenSSH_4.1\r\n<br>5614 2015-03-11 08:29:07 Sendlen : 21<br>5614 2015-03-11 08:29:07 Exp.text: SSH<br>5614 2015-03-11 08:29:07 Exp.len : 3<br>5614 2015-03-11 08:29:07 Exp.ofs : 0<br>5614 2015-03-11 08:29:07 Flags : 1<br>5614 2015-03-11 08:29:07 Port : 22<br>5614 2015-03-11 08:29:07 Name : ssh2<br>5614 2015-03-11 08:29:07 Sendtext: SSH-2.0-OpenSSH_4.1\r\n<br>5614 2015-03-11 08:29:07 Sendlen : 21<br>5614 2015-03-11 08:29:07 Exp.text: SSH<br>5614 2015-03-11 08:29:07 Exp.len : 3<br>5614 2015-03-11 08:29:07 Exp.ofs : 0<br>5614 2015-03-11 08:29:07 Flags : 1<br>5614 2015-03-11 08:29:07 Port : 22<br>5614 2015-03-11 08:29:07 Name : telnet<br>5614 2015-03-11 08:29:07 2015-03-11 08:29:08 Child process 5614 died: Signal 6</span></font><br><br></div>From rrd-status.log<font size="1"><span style="font-family:monospace,monospace"><br>5376 2015-03-11 08:29:06 Opening file /opt/local/xymon/server/etc/rrddefinitions.cfg<br>5376 2015-03-11 08:29:06 Want msg 1, startpos 0, fillpos 0, endpos -1, usedbytes=0, bufleft=2101247<br>5376 2015-03-11 08:29:06 Got 6075 bytes<br>5376 2015-03-11 08:29:06 xymond_rrd: Got message 621101 @@status#621101/sinfctap01|1426033745.004291|172.21.198.73||sinfctap01|svcs|1426035545|green||green|1417272435|0||0||1426033594|sunos|unix/appzones|0|<br>5376 2015-03-11 08:29:06 startpos 6075, fillpos 6075, endpos -1<br>5376 2015-03-11 08:29:06 Transport setup is:<br>5376 2015-03-11 08:29:06 xymondportnumber = 1984<br>5376 2015-03-11 08:29:06 xymonproxyhost = NONE<br>5376 2015-03-11 08:29:06 xymonproxyport = 0<br>5376 2015-03-11 08:29:06 Recipient listed as '172.21.219.192'<br>5376 2015-03-11 08:29:06 Standard protocol on port 1984<br>5376 2015-03-11 08:29:06 Will connect to address 172.21.219.192 port 1984<br>5376 2015-03-11 08:29:06 Connect status is 0<br>5376 2015-03-11 08:29:06 Sent 16 bytes<br>5376 2015-03-11 08:29:06 Read 8192 bytes<br>5376 2015-03-11 08:29:06 Read 7657 bytes<br>5376 2015-03-11 08:29:06 Closing connection<br>5376 2015-03-11 08:29:06 Opening file /opt/local/xymon/server/etc/analysis.cfg<br>5376 2015-03-11 08:29:06 Compiling regex ctrldom[A-D]<br>5376 2015-03-11 08:29:06 Compiling regex infdom[A-B]1<br>5376 2015-03-11 08:29:06 Compiling regex .+dom..<br>5376 2015-03-11 08:29:06 Compiling regex infdom[A-B]1|notesdomD1<br>5376 2015-03-11 08:29:06 Compiling regex [.:]80$<br>5376 2015-03-11 08:29:06 Compiling regex syn.*,%oradom.*<br>5376 2015-03-11 08:29:06 Compiling regex syn.*|oradom.*|SINFV.*<br>5376 2015-03-11 08:29:06 Compiling regex [.:]514$<br>5376 2015-03-11 08:29:06 Compiling regex [.:]3389$<br>5376 2015-03-11 08:29:06 Compiling regex energise.*<br>5376 2015-03-11 08:29:06 Compiling regex [.:]8989$<br>5376 2015-03-11 08:29:06 Compiling regex [.:]80$<br>5376 2015-03-11 08:29:06 Compiling regex [.:]4848$<br>5376 2015-03-11 08:29:06 Compiling regex [.:]8080$<br>5376 2015-03-11 08:29:06 Compiling regex ^/var/crash/.*<br>5376 2015-03-11 08:29:06 Compiling regex ^/var/core/.*<br>5376 2015-03-11 08:29:06 Compiling regex [.:]22$<br>5376 2015-03-11 08:29:06 Compiling regex sinfct.*|domino.*<br>5376 2015-03-11 08:29:06 Compiling regex [.:]80$<br>5376 2015-03-11 08:29:06 Compiling regex [.:]25$<br>5376 2015-03-11 08:29:06 Compiling regex sinfct.*|domino.*|SINFV.*|hermes|sirocco<br>5376 2015-03-11 08:29:06 Compiling regex SINFV.*<br>5376 2015-03-11 08:29:06 Compiling regex .*<br>5376 2015-03-11 08:29:06 Compiling regex ^error.*<br>5376 2015-03-11 08:29:06 Compiling regex ^error.*<br>5376 2015-03-11 08:29:06 Compiling regex .*restart/shutdown.*|.*Citrix<br>5376 2015-03-11 08:29:06 Compiling regex .*<br>5376 2015-03-11 08:29:06 Compiling regex ^warning.*<br>5376 2015-03-11 08:29:06 Compiling regex ^warning.*<br>5376 2015-03-11 08:29:06 Compiling regex .*MetaFrameEvents.*|.*restart/shutdown.*|.*Citrix-Multimedia-Flash.*|.*Printer<br>5376 2015-03-11 08:29:06 Compiling regex .*<br>5376 2015-03-11 08:29:06 Compiling regex ^failure.*<br>5376 2015-03-11 08:29:06 Compiling regex ^failure.*<br>5376 2015-03-11 08:29:06 Compiling regex ^/var/crash/.*<br>5376 2015-03-11 08:29:06 Compiling regex ^/var/core/.*<br>5376 2015-03-11 08:29:06 Compiling regex [.:]22$<br>5376 2015-03-11 08:29:06 Compiling regex ^/var/crash/.*<br>5376 2015-03-11 08:29:06 Compiling regex ^/var/core/.*<br>5376 2015-03-11 08:29:06 Compiling regex [.:]22$<br>5376 2015-03-11 08:29:06 Opening file /opt/local/xymon/server/etc/protocols.cfg<br>5376 2015-03-11 08:29:06 Service list dump<br>5376 2015-03-11 08:29:06 Name : ftp<br>5376 2015-03-11 08:29:06 Sendtext: quit\r\n<br>5376 2015-03-11 08:29:06 Sendlen : 6<br>5376 2015-03-11 08:29:06 Exp.text: 220<br>5376 2015-03-11 08:29:06 Exp.len : 3<br>5376 2015-03-11 08:29:06 Exp.ofs : 0<br>5376 2015-03-11 08:29:06 Flags : 1<br>5376 2015-03-11 08:29:06 Port : 21<br>5376 2015-03-11 08:29:06 Name : ftps<br>5376 2015-03-11 08:29:06 Sendtext: quit\r\n<br>5376 2015-03-11 08:29:06 Sendlen : 6<br>5376 2015-03-11 08:29:06 Exp.text: 220<br>5376 2015-03-11 08:29:06 Exp.len : 3<br>5376 2015-03-11 08:29:06 Exp.ofs : 0<br>5376 2015-03-11 08:29:06 Flags : 5<br>5376 2015-03-11 08:29:06 Port : 990<br>5376 2015-03-11 08:29:06 Name : ssh<br>5376 2015-03-11 08:29:06 Sendtext: SSH-2.0-OpenSSH_4.1\r\n<br>5376 2015-03-11 08:29:06 Sendlen : 21<br>5376 2015-03-11 08:29:06 Exp.text: SSH<br>5376 2015-03-11 08:29:06 Exp.len : 3<br>5376 2015-03-11 08:29:06 Exp.ofs : 0<br>5376 2015-03-11 08:29:06 Flags : 1<br>5376 2015-03-11 08:29:06 Port : 22<br>5376 2015-03-11 08:29:06 Name : ssh1<br>5376 2015-03-11 08:29:06 Sendtext: SSH-2.0-OpenSSH_4.1\r\n<br>5376 2015-03-11 08:29:06 Sendlen : 21<br>5376 2015-03-11 08:29:06 Exp.text: SSH<br>5376 2015-03-11 08:29:06 Exp.len : 3<br>5376 2015-03-11 08:29:06 Exp.ofs : 0<br>5376 2015-03-11 08:29:06 Flags : 1<br>5376 2015-03-11 08:29:06 Port : 22<br>5376 2015-03-11 08:29:06 Name : ssh2<br>5376 2015-03-11 08:29:06 Sendtext: SSH-2.0-OpenSSH_4.1\r\n<br>5376 2015-03-11 08:29:06 Sendlen : 21<br>5376 2015-03-11 08:29:06 Exp.text: SSH<br>5376 2015-03-11 08:29:06 Exp.len : 3<br>5376 2015-03-11 08:29:06 Exp.ofs : 0<br>5376 2015-03-11 08:29:06 Flags : 1<br>5376 2015-03-11 08:29:06 Port : 22<br>5376 2015-03-11 08:29:06 Name : telnet<br>5376 2015-03-11 08:29:06 2015-03-11 08:29:07 Peer at <a href="http://0.0.0.0:0">0.0.0.0:0</a> failed: Broken pipe<br>2015-03-11 08:29:07 Child process 5376 died: Signal 6</span></font><br><br><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On 11 March 2015 at 09:52, Jeremy Laidman <span dir="ltr"><<a href="mailto:jlaidman@rebel-it.com.au" target="_blank">jlaidman@rebel-it.com.au</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class="">On 11 March 2015 at 11:37, Vernon Everett <span dir="ltr"><<a href="mailto:everett.vernon@gmail.com" target="_blank">everett.vernon@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div>And even with --no-cache, I am still getting these corrupted rrd files.<br></div></div></div></div></div></div></div></blockquote><div><br></div></span><div>:-(</div><span class=""><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div></div></div>I tried again with --debug (and --no-cache) and it core dumps.<br><br></div>Here's the backtrace.<br></div></div></div></div></blockquote><div> </div></span><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div>libc.so.1`vfprintf+0xec(6c3d0, 514c0, ffbfb3e8, 0, a0ba4, 33e1c)<span class=""><br>dbgprintf+0xa4(514c0, 0, 51400, 6c3f0, bf, 2ab388)<br>dump_tcp_services+0x74(a0, 1c00, fef37940, 0, 51400, 51400)<br></span></div></div></div></div></blockquote><div><br></div><div>So dump_tcp_services() calls dbgprintf() (both on lib/netservices.c) which in turn calls vprintf() from libc, but with bad parameters. I've had a look through the code in dump_tcp_services() and I don't know enough C to recognize any problems. But it might be useful to know which call to dbgprintf() is causing the problem.</div><div><br></div><div>Does the log file for xymond_rrd show any debug output at all? If so, what's the last line that is shown.</div><div><br></div><div>It might be helpful if you can recompile xymond_rrd with dump_tcp_services() modified. Initially, I would simply try it with "return" added after the first call to dbgprintf(). That is, dump_tcp_services() will output "Service list dump" and return. This might stop the core dumps so that we can get debug output for other parts of the xymond_rrd processing.</div><div><br></div><div>If adding "return" at that point fixes this core dump, more diagnostic lines would be useful to determine what the problem is. For example, there's a global array called svcinfo that is iterated over, but if the array is empty, it might cause the core dump. So adding a line that checks whether the array is empty and displays the result would help to pin this down.</div><div><br></div><div>Note that "svcinfo" appears to be populated from the protocols.cfg file and/or XYMONNETSVCS. Is it possible that your protocols.cfg file is empty, or has some syntax error that causes it to be unparseable? The same for XYMONNETSVCS (in xymonserver.cfg)?</div><span class="HOEnZb"><font color="#888888"><div><br></div><div>J</div><div><br></div></font></span></div></div></div>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature"><span>"Accept the challenges so that you can feel the exhilaration of victory"</span><div><span>- General George Patton</span></div></div>
</div>