[Xymon] disk graph page limits total file systems

Wed May 25 20:40:43 CEST 2016

JC,

I think I'm starting to see a pattern emerge, and a theory develop, here.
Hope everyone is able to follow this... here goes:

I think there may potentially be a disconnect between how the disk page
determines how many filesystems SHOULD be graphed and the number of RRD
files that are available TO BE graphed.  I think the reason the trends page
seems to work OK is it just graphs all data that it has available without
condition.  It seems like the disk page determines 1) the number of
filesystems to graph and,  2) based on that number, the number of
filesystems per image.  These numbers seem to be determined BEFORE it
generates the HTML that produces the link HREFs and image SRCs.  It then
seems to produce just enough graphs to satisfy the predetermined number,
plus enough to satisfy the predetermined "multiple",

The predetermined number of filesystems seems to come from the number of
filesystems reported in the previous message from the client.   I believe
the assumption was made those numbers should always match.  And for the
most part they do.  It's not everyday sys admins remove filesystems from
their systems.  Until now, it may not have been so easy to spot.  It's a
little more obvious to me, being primarily an AIX administrator.  We have a
daily process that creates an "alt_disk_copy" of our rootvg so that we
always have a hot backup of the OS.  This process causes a lot of transient
filesystems to be created.  Those filesystems get reported and recorded
during the brief window that this process is running.  On closer
examination of my AIX systems, it is not just the one with 85 filesystems
getting truncated... it's all of them.
When I view HTML source on the disk page, I see that just ahead of the HTML
code that displays the graph images and links, there is an HTML comment
line: "" Where x equals the number of filesystems that
were reported in the previous message from the client.  (Count number of
lines, excluding header, from [df] section.) I went through each of my
systems, Linux and AIX, and determined that to always be the case.  There
must also be some range at which it determines the number "y" that
determines how many filesystems to display on each graph.  It seems like
that number is x<80, y=4 and x>=80, y=5.  (If y changes to 6 at some point,
I haven't done enough testing to determine where that threshold is.)

The request to showgraph.cgi includes the parameters first=z and count=y.
If there are no more RRD files to graph, it stops and the graph shows fewer
filesystems than the count parameter.  But, if you have a situation where
you have more data available than the predetermined number of filesystems,
it will continue to graph them.

On the system that previously seem limited to 85 file systems, I modified
the "hobbitclient-$os.sh" script and grepped out a certain number of file
systems.  After doing this, I had 77 filesystems reported.  That number was
reflected in the "linecount=" HTML comment, and I also began seeing 4
filesystems per graph (instead of 5, previously) and 20 graphs being
displayed (instead of 17, previously) for a total of 80 filesystems being
graphed.  It graphed 80 because it still had enough data from RRD files to
round out the last graph.  Also, the filesystems that were grepped out of
the message from the client, were still graphed.

I also went back and checked my Linux systems; the ones where I added 100
filesystems.  On those systems, I created enough filesystems to push past
that 85 filesystem "limit".  Since those all graphed successfully, I had
previously thought that it was the difference between AIX and Linux.  That
no longer seems to be the case.  Now that I have removed all of those test
file systems, and since it's only reporting 10 filesystems, only 10
filesystems are being graphed.  File systems like /, /boot, and /home are
graphed... but the test ones that I removed, are still being graphed, and
filesystems that you would expect to see at the end alphabetically,
(e.g. /usr, /var, /opt, /tmp, etc) are not displayed.

A lot of speculation, I realize that, but the theory seems to fit reality
in all cases.  I haven't examined the code to prove it out since, as I've
said before, my C skills are rubbish.  But if my theory proves to be true,
the suggestion for improvement that I would offer is, make sure that at
least every file system from the most recent message is represented, plus
any additional file systems that might have data available in the time
period requested; between "graph_start" and "graph_end".

Erik D. Schminke | Associate Systems Programmer
Hormel Foods Corporation | One Hormel Place | Austin, MN 55912
Phone: (507) 434-6817
edschminke at hormel.com | www.hormelfoods.com

From:	"J.C. Cleaver" <cleaver at terabithia.org>
To:	EDSchminke at Hormel.com
Cc:	"Xymon Mailing List" <xymon at xymon.com>
Date:	05/23/2016 10:32 PM
Subject:	Re: [Xymon] disk graph page limits total file systems

Hi Erik,

This actually helps a great deal, as it implies there's a distinction in
parsing code ... and potentially not an issue on the display side at all
(which I've been pouring over with little success).

Can you confirm whether the RRD files themselves are being properly
updated for both the AIX and Linux systems? (It might help to disable
caching in xymond_rrd during this process, if your system has enough space
I/O capacity.) In theory all partitions that are coming in should have
their .rrd files updated continually, but if there's a parsing issue then
that might explain one aspect of the failure.

Alternatively, can you try adding and removing partition values in the
client report and see if going above and below the 85-parition value
reliably enables the 86th?

It might be helpful to manually edit the xymonclient-$OS.sh script to grep
out (or include additional) lines of the 'df' output.

Can you also confirm that the remainder of the client report
(CPU/memory/etc.) is being handled OK, even on the AIX system?

So far I've been unable to duplicate this, but I was primarily testing on
x86_64 Linux VMs.

Regards,
-jc