<div dir="ltr">Thanks for your assistance Jeremy.<div>Quite a bit to digest.</div><div><br></div><div>Everything in the <span>xymon</span>.out file I am collecting looks exactly like I would expect it to look.</div><div>There is a line, prefixed by @@data#<sequence> (or at least I think it's a sequence number),followed by pipe-separated data that looks like the host name, time stamp, <span>IP</span> address, an empty field, host name again, the rrd-file prefix, another blank field and the last field is other.</div><div>I then get a blank line, and the data looks like what I am trying to send.</div><div><br></div><div><br></div><div>Looks like we might need to check with <span>JC</span> for more on that <span>GOCLIENT</span> thing.</div><div>I just find it odd that it happened about the same time as the corruption.</div><div>I haven't seen it again today, and haven't seen any other corruption either.</div><div><br></div><div>At this site, we are running <span>Xymon</span> 4.3.12, but I have seen similar behaviour (although not to such an extent) elsewhere with 4.3.17, and I think I also saw it with 4.3.18, but I no longer have access to that site.</div><div><br></div><div>I am not seeing any lost data points in the other graphs. But that could be difficult to spot.</div><div>Will run a few <span>rrdtool</span> dumps, and look for gaps at that <span>timestamp</span>. Let you know what I find.</div><div><br></div><div>As for the --debug option, it caused <span>xymond</span>_<span>rrd</span> to crash and burn, dumping cores as we go.</div><div>It gets ugly.</div><div>Earlier in this thread, John Thurston mentioned this behaviour too.</div><div>It also creates a red <span>xymond</span>_<span>rrd</span> button on the <span>xymon</span> server, but the results are not overly helpful.</div><div><h3 style="color:rgb(216,216,191);font-family:'Times New Roman';background-color:rgb(0,0,0)">- Program crashed</h3><h3 style="color:rgb(216,216,191);font-family:'Times New Roman';background-color:rgb(0,0,0)"><span style="font-family:arial,sans-serif;font-size:small;font-weight:normal">Fatal signal caught!</span></h3></div><div><br></div><div>Don't think it started after an upgrade.</div><div>Something I did notice, the problem appears to be limited to data only, used to display graphs in trends.</div><div>I am not seeing this for data when there is a status and data component.</div><div>Or at least I haven't seen it yet.</div><div><br></div><div>What are the implications of running with <span style="font-size:12.8000001907349px">"--no-cache"?</span></div><div><span style="font-size:12.8000001907349px">I have implemented this by adding "--no-cache" but if it's going to have a long-term impact, I don't want to leave it that way indefinitely.</span></div><div><span style="font-size:12.8000001907349px"><br></span></div><div><span style="font-size:12.8000001907349px">Regards</span></div><div><span style="font-size:12.8000001907349px">Vernon</span></div><div><span style="font-size:12.8000001907349px"><br></span></div><div><span style="font-size:12.8000001907349px"><br></span></div><div class="gmail_extra"><br></div><div class="gmail_extra"><br><div class="gmail_quote">On 4 March 2015 at 14:03, Jeremy <span>Laidman</span> <span dir="ltr"><<a href="mailto:jlaidman@rebel-it.com.au" target="_blank"><span>jlaidman</span>@rebel-it.com.<span>au</span></a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span>On 4 March 2015 at 12:40, Vernon Everett <span dir="ltr"><<a href="mailto:everett.vernon@gmail.com" target="_blank">everett.vernon@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div>Here's what I ran, with error output.<br></div></div><div><font size="1"><span style="font-family:monospace,monospace">./xymoncmd xymond_channel --channel=data --filter=e-series cat > /var/tmp/xymon.out<br>2015-03-04 08:45:22 Using default environment file /opt/local/xymon/server/etc/xymonserver.cfg<br>2015-03-04 08:45:58 Peer not up, flushing message queue<br>2015-03-04 09:05:21 Gave up waiting for GOCLIENT to go low.</span></font><br><br></div><div>What is that GOCLIENT thing?<br></div></div></div></div></div></blockquote><div><br></div></span><div>From what I can understand, it's a semaphore shared between xymond and all of the xymond_channel instances. When there are several channel readers, they all get sent the message address, and as each one accepts the message, she decrements GOCLIENT. When GOCLIENT is zero, it means all readers have received (and probably copied) the message, and the memory can be freed. Each reader waits until GOCLIENT goes back to zero before waiting for the next message.</div><div><br></div><div>There's a timeout of 1 second that xymond_channel waits for GOCLIENT to go back to zero. If the time is exceeded in a channel reader, it means another reader is taking too long to handle a message, and so the first reader gives up, logs the error you saw, and carries on with the next message loop. I'm not sure if this is a sign of trouble. Or it might be normal when you're running your own instance of xymond_channel. Or it might be a side-effect of the "cat" command blocking when writing to your output file due to a high message rate and contention on whatever filesystem has /var/tmp/.</div><div><br></div><div>There's a description of how GOCLIENT works in the file new-daemon.txt, in the source code.</div><span><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div>In the output file, /var/tmp/xymon.out from <br></div><div><font size="1"><span style="font-family:monospace,monospace">./xymoncmd xymond_channel --channel=data --filter=e-series cat > /var/tmp/xymon.out</span></font><br></div><div>there is no mention of the subversion or energise stuff either.<br></div></div></div></div></div></blockquote><div><br></div></span><div>Does it have mention of the correct data set names? We can't draw any conclusions if it's not collecting the data we expect.</div><div><br></div><div>Did any of the RRD files skip an update at the time the new rogue files were created? Do these files match up with entries in xymon.out? Or anything interesting at the same time as the rogue entries were created?</div><div><br></div><div>If you're seeing correct entries in xymon.out, but not the bogus entries, then I'm inclined to agree that xymond_rrd is at fault, and is possibly using memory it's not supposed to. I wonder if running xymond_rrd with "--no-cache" might have an effect. Obviously, it's better if you can cache updates to the RRD files, but it might narrow down the region of code that's responsible.</div><div><br></div><div>This is not conclusive. It's possible that when you have two instances of xymond_channel, only one is corrupting data names, and it just so happened that it was the one being used by xymond_rrd. Could be that another time you would see your extra reader getting the bogus entries. That's the problem with using a second instance for analysis, rather than somehow getting the analysis happening on the one that writes to the RRD files.</div><div><br></div><div>On the other hand, if you ran two instances of xymond_rrd, both on the same data channel, and if both instances create the bogus RRD files, then you know that you can probably use the second instance to narrow down the fault, without impacting the creation of RRD files for real work.</div><div><br></div><div>Are you still running xymond_rrd with "--debug"? Did this show anything interesting when the bogus RRD files were created?</div><div><br></div><div>What version of Xymon are you running? Did this start happening after an upgrade? I wonder if it's a bug with some versions but not others.</div><span><font color="#888888"><div><br></div><div>J</div><div><br></div></font></span></div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div><span>"Accept the challenges so that you can feel the exhilaration of victory"</span><div><span>- General George Patton</span></div></div>
</div></div>