Unfortunately, no, I can't do this as our Hobbit server monitors production machines. The data directory for the rrd files are SAN-mounted, and we haven't had disk corruption issues before with this type of setup.
<br><br>The strange thing is, this only started within the past week, and unfortunately it seems to be spreading to more and more RRD graphs.<br><br><div class="gmail_quote">On Nov 29, 2007 3:41 PM, Josh Luthman <<a href="mailto:josh@imaginenetworksllc.com">
josh@imaginenetworksllc.com</a>> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Can you do a dd if=/dev/sda of=/dev/null from the disk in which the stuff is stored? If it is so random I'm curious to see if the fs is having problems. I have my money on a bug in the software or bad disk/fs/controller.
<div><div></div><div class="Wj3C7c"><br><br><div><span class="gmail_quote">On 11/29/07, <b class="gmail_sendername">Gary Baluha</b> <<a href="mailto:gumby3203@gmail.com" target="_blank">gumby3203@gmail.com</a>> wrote:
</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
I don't know how many hosts are affected, percentage wise, but it's definitely not every host. And for the hosts having the problem, it's not even the same graphs that are having the problem.<div><span>
<br><br><div class="gmail_quote">
On Nov 29, 2007 3:11 PM, Josh Luthman <<a href="mailto:josh@imaginenetworksllc.com" target="_blank">josh@imaginenetworksllc.com</a>> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Same OS at home?<br><br>Not sure if you mentioned this or not but does that weird value show up in all RRD graphs or just a few hosts?<div><div></div><div><br><br><div><span class="gmail_quote">On 11/29/07,
<b class="gmail_sendername">Josh Luthman
</b> <<a href="mailto:josh@imaginenetworksllc.com" target="_blank">josh@imaginenetworksllc.com</a>> wrote:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Do they monitor the same devices? I think there has to be some similarity between the two as they had the same problem at the same time (though this isn't 100%, it's logically the first place to look). Hardware isn't of much concern here as they don't communicate and the chances of both servers going bad on the same date is simply astronomical.
<br><br>Are there any kind of auto updating services running on them?<div><span><br><br><div><span class="gmail_quote">On 11/29/07, <b class="gmail_sendername">Gary Baluha</b> <<a href="mailto:gumby3203@gmail.com" target="_blank">
gumby3203@gmail.com
</a>> wrote:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">We only have two Hobbit servers, and it is affecting both machines. No, these two Hobbit machines do _not_ communicate with each other in any way.
<div><span><br><br><div class="gmail_quote">On Nov 29, 2007 2:33 PM, Josh Luthman <
<a href="mailto:josh@imaginenetworksllc.com" target="_blank">josh@imaginenetworksllc.com</a>> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Is this problem
<span style="font-weight: bold;">not </span>showing up on another Hobbit server? Do the two Hobbit servers with this problem communicate at all (share data/SNMP traffic/etc)?<br><br><div><div><span class="gmail_quote">
On 11/29/07, <b class="gmail_sendername">Gary Baluha</b> <<a href="mailto:gumby3203@gmail.com" target="_blank">gumby3203@gmail.com</a>> wrote:</span></div><div>
<div></div><div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<span>On Nov 29, 2007 12:01 PM, Josh Luthman <<a href="mailto:josh@imaginenetworksllc.com" target="_blank">josh@imaginenetworksllc.com</a>> wrote:<br>
</span><div class="gmail_quote"><span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
This is completely beyond my knowledge, but the first place I would look at is any hardware problems, any recent changes (obviously =) and the similarities between those two Hobbit servers issues.<div><div></div></div></blockquote>
</span><div><br>That's the thing, there aren't any similarities between these two machines. They are different hardware, different OS, different network segment, and different hosts being monitored.<br><br>There were some recent changes in the past month to one of the hobbit servers, with a bunch of custom RRD graphs added. But this wasn't done on the other hobbit server. The only thing changed on the other hobbit server is more html web checks added; nothing out of the ordinary.
<br></div></div><br>
</blockquote></div></div></div><br><br clear="all"><div><div></div><div><br>-- <br>Josh Luthman<br>Office: 937-552-2340<br>Direct: 937-552-2343<br>1100 Wayne St<br>Suite 1337<br>Troy, OH 45373<br><br>Those who don't understand UNIX are condemned to reinvent it, poorly.
<br>--- Henry Spencer
</div></div></blockquote></div><br>
</span></div></blockquote></div><br><br clear="all"><br>-- <br>Josh Luthman<br>Office: 937-552-2340<br>Direct: 937-552-2343<br>1100 Wayne St<br>Suite 1337<br>Troy, OH 45373<br><br>Those who don't understand UNIX are condemned to reinvent it, poorly.
<br>--- Henry Spencer
</span></div></blockquote></div><br><br clear="all"><br>-- <br>Josh Luthman<br>Office: 937-552-2340<br>Direct: 937-552-2343<br>1100 Wayne St<br>Suite 1337<br>Troy, OH 45373<br><br>Those who don't understand UNIX are condemned to reinvent it, poorly.
<br>--- Henry Spencer
</div></div></blockquote></div><br>
</span></div></blockquote></div><br><br clear="all"><br>-- <br>Josh Luthman<br>Office: 937-552-2340<br>Direct: 937-552-2343<br>1100 Wayne St<br>Suite 1337<br>Troy, OH 45373<br><br>Those who don't understand UNIX are condemned to reinvent it, poorly.
<br>--- Henry Spencer
</div></div></blockquote></div><br>