[Xymon] diskstat.sh/RRD oddity

[Steve Holmes]

This is just a comment on an oddity with respect to diskstat.sh and RRD.

We make pretty heavy use of the diskstat.sh script, which I believe I
downloaded from xymonton. When I installed it I used the standard
clientlaunch.cfg stanza for the configuration and everything worked great.

I was called to task today because we have been having some disk io issues
on the RHEL VMs and someone was looking at the trend graphs for some
servers to see if there was anything they could learn and they noticed that
beginning at about 4pm local time on Monday the graphs for the number of
sectors written per second on a couple of file systems on several VMs
jumped from the 10 to 20 range to the 300 to 340 range and stayed there.
The graph for number of disk writes per second had a corresponding jump up
to about 40 or 50 from close to zero.

In analyzing the data I discovered that the file system that was displaying
this behavior is the same file system to which the diskstat.sh script is
writing its temp files. It appears that for some reason, starting at 4pm on
Monday the 5 minute test interval and the 5 minute average for RRD got in
sync and all it was seeing was the data point that corresponded to its own
writing activity and RRD was using it for the entire 5 minute average (of
course, that's what RRD does).

I 'fixed' it by changing the test interval to something less than 5
minutes. I tried 2, 3, and 4 minutes and they all had the effect of
reducing the data in the plot back to the expected level, i.e. to the level
it was before 4pm on Monday.

The mystery remains why it suddenly started seeing and using its own disk
activity at the same time on several different servers.

Steve Holmes
ITaP/Purdue University

-- 
If they give you ruled paper, write the other way. -Juan Ramon Jimenez,
poet, Nobel Prize in literature (1881-1958)

I prayed for freedom for twenty years, but received no answer until I
prayed with my legs. -Frederick Douglass, Former slave, abolitionist,
editor, and orator (1817-1895)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xymon.com/pipermail/xymon/attachments/20120328/fb1ae2ea/attachment.html>