[Xymon] "ports" RRD graph is often flaky.

[Jeremy Laidman]

Shawn

On Wed, Mar 23, 2016 at 9:47 AM Shawn Heisey <hobbit at elyograg.org> wrote:

> I see several graphs like this in xymon (4.3.23), where at least one of
> the port graphs (configured in analysis.cfg) has lots of data points
> missing:
>
> https://www.dropbox.com/s/l483lilzh5n610k/flaky-xymon-port-graph.png?dl=0
>
> Any idea how I can fix this?  As you can see, the red line for http has
> data for the entire graph, while the blue line for ajp is very spotty.
> I just added the last two monitors less than an hour ago, so I don't
> have history on those yet.
>
> This is not unique to this host.  As far as I can tell, the port
> listings coming back from the client are always good, even when the
> graph has no line.
>
> Thanks,
> Shawn
>

Firstly, I'm assuming you have setup port tracking with "PORT bla bla
STATE=ESTABLISHED TRACK=ajp" or similar in analysis.cfg.  It might be
helpful to provide the relevant config section from there.

I suppose there are three likely points of failure in the process of
getting port data into the graphs.  The first one is where the port data is
received by Xymon and passed to the RRD parser xymond_rrd.  The second is
where the port lists are analysed to get the numbers.  The third is where
the numbers are added to the RRD file.

If your "ports" tests show "red" or "clear" then that would suggest the
first part is failing.  If there are problems with the second or third
parts, you may see useful messages logged in the rrd-status.log file.

You could check to see if the port lists are being sent to the xymond_rrd
process that handles status messages (and writes to the rrd-status.log
file).  If they are, then you'll know there's probably something amiss with
the RRD analysis or update systems.  Otherwise, if there are gaps in the
messages being sent to the status channel, then it indicates that the RRD
updating is probably fine.

You can take a feed of the port status messages coming out of the status
channel, so that you see everything that gets sent to xymond_rrd.  First
switch to the xymon user, and then run xymoncmd to setup your environment.
Then run a command such as the following:

$ xymond_channel --channel=status --filter='\|fremont\|ports\|' cat

This will only display status channel messages for the host "fremont" and
for the test name "ports". I'm using "cat" as the worker process, but you
can substitute this with a grep, awk or sed command to reduce screen output
as desired, or even write a script to parse the output.  If you only want
to know that a status message has been sent (and don't care about its
contents), you could replace "cat" with "grep '^@@'", but for a first pass,
"cat" works just fine.

You can extend this idea to see what xymond_rrd is doing when it receives
these messages, by running your own copy of xymond_rrd, with debugging
enabled.  Something like so:

$ mkdir /tmp/myrrd
$ xymond_channel --channel=status --filter='\|fremont\|ports\|' xymond_rrd
--rrddir=/tmp/myrrd/ --debug

This produces lots of output as xymond_rrd loads its configuration when it
receives the first message, but then the output rate drops off to a few
lines for each message.

Due to buffering of the RRD file updates, it's tricky to debug whether
xymond_rrd is sending updates to the RRD files.

Cheers
Jeremy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xymon.com/pipermail/xymon/attachments/20160323/dc6aed7b/attachment.html>