<div dir="ltr">Hi Jeremy<div><br></div><div>Added some debug code to my script. Here's an extract.</div><div><div><div><font face="monospace, monospace"> DATA=$(cat $TMPFILE.drvperf | awk '{ print $1" : "$2 }') # Current IO latency</font></div><div><font face="monospace, monospace"> $XYMON $XYMSRV "data $ENAME.e-series-dcuriolat $(echo; echo; echo "$DATA"; echo)"</font></div><div><font face="monospace, monospace">echo $XYMON $XYMSRV "data $ENAME.e-series-dcuriolat $(echo; echo; echo "$DATA"; echo)"</font></div><div><font face="monospace, monospace"> DATA=$(cat $TMPFILE.drvperf | awk '{ print $1" : "$3 }') # Max IO latency</font></div><div><font face="monospace, monospace"> $XYMON $XYMSRV "data $ENAME.e-series-dmaxiolat $(echo; echo; echo "$DATA"; echo)"</font></div><div><font face="monospace, monospace">echo $XYMON $XYMSRV "data $ENAME.e-series-dmaxiolat $(echo; echo; echo "$DATA"; echo)"</font></div><div><font face="monospace, monospace"> DATA=$(cat $TMPFILE.drvperf | awk '{ print $1" : "$3 }') # Avg IO latency</font></div><div><font face="monospace, monospace"> $XYMON $XYMSRV "data $ENAME.e-series-davgiolat $(echo; echo; echo "$DATA"; echo)"</font></div><div><font face="monospace, monospace">echo $XYMON $XYMSRV "data $ENAME.e-series-davgiolat $(echo; echo; echo "$DATA"; echo)"</font></div></div><div><span style="font-family:monospace,monospace"> </span><br></div></div><div><font face="arial, helvetica, sans-serif">And I managed to get a couple of bizarre data files.</font></div><div><div style="font-family:monospace,monospace">e-series-dcuriolat,icmpOutParmProbs.rrd</div><div style="font-family:monospace,monospace">e-series-dcuriolat,icmpOutRedirects.rrd</div><div style="font-family:monospace,monospace"><div>e-series-dcuriolat,ipv6InTruncatedPkts.rrd</div><div>e-series-dcuriolat,ipv6OutFragFails.rrd</div></div><div style="font-family:monospace,monospace"><div>e-series-dcuriolat,UDP_udpInDatagrams.rrd</div><div>e-series-dcuriolat,udpInCksumErrs.rrd</div></div><div style="font-family:monospace,monospace"><br></div><div><font face="arial, helvetica, sans-serif">And if I grep in my log file for icmp or any of those terms, I come up with nothing.</font></div><div><font face="arial, helvetica, sans-serif">So I am guessing it's not coming from the client.</font></div><div><font face="arial, helvetica, sans-serif"><br></font></div><div><font face="arial, helvetica, sans-serif">I want to try the snoop, but this client script is running on the server, as a client script.</font></div><div><font face="arial, helvetica, sans-serif">It collects data from a bunch of NetApp E-series devices, and sends it to the server in the normal way.</font></div><div><font face="arial, helvetica, sans-serif">So you can imagine what the snoop data is going to look like.</font></div><div><font face="arial, helvetica, sans-serif">But I will give it a go, and see if there is something in it.</font></div><div><font face="arial, helvetica, sans-serif"><br></font></div><div><font face="arial, helvetica, sans-serif">As for debugging the rrd tasks, John was right.</font></div><div><font face="arial, helvetica, sans-serif">Adding --debug to the rrd config causes it to crash.</font></div><div><font face="arial, helvetica, sans-serif">Then I just het heaps of this.</font></div><div><div><font face="monospace, monospace">2015-02-25 11:31:07 Peer not up, flushing message queue</font></div><div><font face="monospace, monospace">2015-02-25 11:31:07 Peer not up, flushing message queue</font></div><div><font face="monospace, monospace">2015-02-25 11:31:07 Peer not up, flushing message queue</font></div><div style="font-family:arial,helvetica,sans-serif">And the occasional </div><div><div><font face="monospace, monospace">19073 2015-02-25 11:31:14 2015-02-25 11:31:15 Child process 19073 died: Signal 6</font></div></div><div style="font-family:arial,helvetica,sans-serif"><br></div></div><div><font face="arial, helvetica, sans-serif">But I think I am reasonably happy that the strange data isn't coming from the client script.</font></div><div><font face="arial, helvetica, sans-serif">Martin Flemming is a list member in Germany (think) who is helping me test this script.</font></div><div><font face="arial, helvetica, sans-serif">I will ask him if he's seeing the same issues. If not, I think we can rule out the script.</font></div><div><br></div><div><font face="arial, helvetica, sans-serif">Regards</font></div><div><font face="arial, helvetica, sans-serif">Vernon</font></div><div style="font-family:monospace,monospace"><br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On 24 February 2015 at 14:26, Jeremy Laidman <span dir="ltr"><<a href="mailto:jlaidman@rebel-it.com.au" target="_blank">jlaidman@rebel-it.com.au</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>I'm assuming you've checked your debug output from your script to see if the $TEMPFILE.* file contents look OK.</div><div><br></div>Perhaps run your own instance of "xymond_channel --channel=data" to capture the messages as they come from xymond to xymond_rrd. This will generate a lot of output, so you'll want to use "--filter" and perhaps "grep" to trim it down.<div><br></div><div>You could also run snoop/tcpdump at the same time and try to capture the data message as it arrives at your Xymon server. If you have lots of Xymon traffic it might be better to do so on the client side.</div><div><br></div><div>The trick is to get a snapshot at the time that the RRD file is created, without collecting so much data that you run out of disk! So doing things like this:</div><div><br></div><div>while true; do tcpdump -w dump.out -n -c 10000 dest port 1984 and host blabla; gzip dump.out; mv dump.out.gz dump.out-`date +%s`; done</div><div><br></div><div>This will capture 10k of packets at a time, then compress and rotate.</div><div><br></div><div>You can also run xymond in a host-specific debug mode, by appending "--dbghost=HOSTNAME". That will spit out all the traffic into /tmp/xymond.dbg for analysis. Again, you might need to periodically rotate that file and signal xymond to re-open output files (I'm guessing a HUP signal might do this, or just kill the process and have xymonlaunch restart it).</div><div><br></div><div>The path the data take would be:</div><div><br></div><div>[script] -> [xymon client] -> [TCP/1984] -> [xymond] -> [xymond_channel] -> [xymond_rrd] -> [rrd file]</div><div><br></div><div>What we want to do is to watch the traffic/messages to determine which of these components is causing the problem. My first step would be to try to isolate whether it's a client or server problem, hence watching the traffic with tcpdump/snoop. If the traffic is transmitted over the wire in the correct form, then I'd look at what xymond gives to xymond_channel. And so on. Once we can identify the process that creates the phantom entity, we can look for the root cause and then work-arounds/solutions.<span class="HOEnZb"><font color="#888888"><br></font></span></div><span class="HOEnZb"><font color="#888888"><div><br></div><div>J</div><div><br></div></font></span></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On 24 February 2015 at 16:46, Vernon Everett <span dir="ltr"><<a href="mailto:everett.vernon@gmail.com" target="_blank">everett.vernon@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div><div>I am getting those sporadic .rrd files in spades. :-(<br></div><div>Sometimes, only a single data point in the file. But enough files, and your graphs start to look like crap.<br><br></div>Tomorrow I am off to a client where it's happening all the time.<br></div>What can I send you to assist with investigating?<br><br></div>I am trying to figure out if it's a bug in Xymon, or a bug in my script.<br></div>So far I have no evidence to support it being either.<br></div><br></div>Regards<span><font color="#888888"><br></font></span></div><span><font color="#888888">Vernon<br><br><div><div><div><br><br></div></div></div></font></span></div><div class="gmail_extra"><div><div><br><div class="gmail_quote">On 24 February 2015 at 13:14, Jeremy Laidman <span dir="ltr"><<a href="mailto:jlaidman@rebel-it.com.au" target="_blank">jlaidman@rebel-it.com.au</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span>On 14 November 2014 at 14:43, Vernon Everett <span dir="ltr"><<a href="mailto:everett.vernon@gmail.com" target="_blank">everett.vernon@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div><div><div>Am busy trying to investigate a curious problem with rrd graphs, and I stumbled on something else I don't understand, and was hoping somebody out there could help.<br></div><br></div>As part of my investigation, I added --debug to the [rrdstatus] and [rrddata] entries on the server tasks.cfg<br></div>And the logs started showing heaps of the message<br><span style="font-family:monospace">2014-11-14 10:41:36 Peer not up, flushing message queue</span><br></div>What is that? <br></div>It doesn't look right to me.<br></div></div></div></div></blockquote><div><br></div></span><div>It's usually normal. See Henrik's response to a similar question:</div><div><br></div><div><a href="http://lists.xymon.com/archive/2014-April/039461.html" target="_blank">http://lists.xymon.com/archive/2014-April/039461.html</a></div><span><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div>Except every now and then, I get something like<br></div><div><span style="font-family:monospace">zmem,c2t0d1.rrd</span><br></div><div></div></div></div></div></div></div></div></blockquote><div> </div></span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div>Has anybody seen anything like this?<br></div></div></div></div></div></div></div></blockquote><div><br></div><div>Yes. It's puzzling, but rare enough that I haven't had time to investigate.</div><span><font color="#888888"><div><br></div><div>J</div><div><br></div></font></span></div></div></div>
</blockquote></div><br><br clear="all"><br></div></div><span>-- <br><div><span>"Accept the challenges so that you can feel the exhilaration of victory"</span><div><span>- General George Patton</span></div></div>
</span></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><span>"Accept the challenges so that you can feel the exhilaration of victory"</span><div><span>- General George Patton</span></div></div>
</div>