On Nov 30, 2007 10:53 AM, Hubbard, Greg L <<a href="mailto:greg.hubbard@eds.com">greg.hubbard@eds.com</a>> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div>
<div dir="ltr" align="left"><span><font color="#0000ff" face="Arial" size="2">Gary,</font></span></div>
<div dir="ltr" align="left"><span><font color="#0000ff" face="Arial" size="2"></font></span> </div>
<div dir="ltr" align="left"><span><font color="#0000ff" face="Arial" size="2">This is pretty hard to decipher from
"afar".</font></span></div>
<div dir="ltr" align="left"><span><font color="#0000ff" face="Arial" size="2"></font></span> </div>
<div dir="ltr" align="left"><span><font color="#0000ff" face="Arial" size="2">I think I remember you saying that when you dump the data
it is always okay?</font></span></div></div></blockquote><div><br>Actually, it turns out this is not true. The rrd file does indeed have the bad data. I just didn't notice it before, but now that it appears to be getting worse, it is quite obvious to see the bad data.
<br><br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div><div dir="ltr" align="left"><span><font color="#0000ff" face="Arial" size="2">
</font></span></div>
<div dir="ltr" align="left"><span><font color="#0000ff" face="Arial" size="2">Some wild thoughts:</font></span></div>
<div dir="ltr" align="left"><span><font color="#0000ff" face="Arial" size="2"></font></span> </div>
<div dir="ltr" align="left"><span><font color="#0000ff" face="Arial" size="2">a) could there be two different processes updating the same
RRD files?</font></span></div>
</div></blockquote><div><br>I don't believe so. The strange thing is, all of the graphs that become corrupted have the exact same large number that is being input into the rrd data files.<br> </div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div>
<div dir="ltr" align="left"><span><font color="#0000ff" face="Arial" size="2">b) are all servers using the same version of
rrdtool?</font></span> </div></div></blockquote><div><br>No. One is running 1.2.23, the other 1.2.26. Both have the problem.<br> </div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div><div dir="ltr" align="left"></div>
<div dir="ltr" align="left"><span><font color="#0000ff" face="Arial" size="2">c) are the hobbitgraph files okay? I have proven to
my satisfaction that hobbitgraph definition errors can make the graphs act
funny.</font></span></div>
</div></blockquote><div><br>They haven't changed since before the graphs were having this problem.<br> </div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div>
<div dir="ltr" align="left"><span><font color="#0000ff" face="Arial" size="2">d) if this stuff is on a SAN, can it be moved to local
storage?</font></span></div>
</div></blockquote><div><br>It is on the SAN on one of the machines, and locally on the other. I was thinking of temporarily moving the data directory and have Hobbit regenerate all the data from scratch. I'm trying to avoid this, since that would mean losing a year's worth of trend data that has proven itself very useful. Still, if it helps me narrow down the problem, I'll consider this (and move the data back once I get my answer).
<br> </div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div>
<div dir="ltr" align="left"><span><font color="#0000ff" face="Arial" size="2">I am just "fishing." Sometimes, when I am at my wit's
end, I just change SOMETHING to see if it makes a difference. Even WORSE can
help get me started.</font></span></div>
<div dir="ltr" align="left"><span><font color="#0000ff" face="Arial" size="2"></font></span> </div>
<div dir="ltr" align="left"><span><font color="#0000ff" face="Arial" size="2">GLH</font></span></div><br>
<blockquote style="margin-right: 0px;">
<div dir="ltr" align="left" lang="en-us">
<hr>
<font face="Tahoma" size="2"><div class="Ih2E3d"><b>From:</b> Gary Baluha [mailto:<a href="mailto:gumby3203@gmail.com" target="_blank">gumby3203@gmail.com</a>]
<br></div><b>Sent:</b> Friday, November 30, 2007 9:25 AM<div class="Ih2E3d"><br><b>To:</b>
<a href="mailto:hobbit@hswn.dk" target="_blank">hobbit@hswn.dk</a><br><b>Subject:</b> Re: [hobbit] strange graph behavior - random
machines & graphs<br></div></font><br></div>
<div><div></div><div class="Wj3C7c"><div></div>Now this appears it is becoming a more serious problem. It
seems more and more graphs are starting to be affected, and I still have no
explanation for what is going on here. It also seems that almost any new
graph that is created (such as if I delete/rename/move an existing .rrd file),
it immediately starts off being corrupted. :-( <br><br>
<div class="gmail_quote">On Nov 28, 2007 10:08 AM, Gary Baluha <<a href="mailto:gumby3203@gmail.com" target="_blank">gumby3203@gmail.com</a>> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div>I have recently noticed a strange thing happening with some of the rrd
graphs generated by Hobbit. When you look at the graph, it looks as
though the rrd data is one one format (gauge), but the graph is generating
it in a different format (derive). I can't seem to find any pattern to
the hosts or tests that are exhibiting this strange behavior, and it is only
happening on a handful of graphs. I have attached a picture of one of
these graphs, since I'm not really sure how to describe it. Note the
huge numbers displayed on the curr/min/avg/max line. </div>
<div> </div>
<div>Any idea what's going on here? When I dump the RRD file manually,
everything looks okay. I'm running Hobbit 4.2.0 with the 2007-02-09
allinone patch (I believe the latest). This has only happened in the
past few weeks, though when exactly it started, I don't know. Any
ideas? </div></blockquote></div><br></div></div></blockquote></div>
</blockquote></div><br>