[Xymon] Graph lines for disk and memory showing gaps

[Jeremy Laidman]

Barrie

This is quite interesting. On the one hand, most of the gaps in the lines
don't line up with gaps in other lines. On the other hand, on the graph
with four lines, the red and pink lines appear to have gaps in the same
place 100% of the time. And the green line only appears when the blue line
appears. It's as if there are two concurrent problems that are masking each
others' symptoms.

Yes, you can run xymond_rrd with the "--debug" switch, to enable more
output in the log file.

Rather than re-configuring the parameters for xymond_rrd in tasks.cfg and
restarting, you can run a second copy of it, which runs independently of
the main one, creating and updating its own copy of the RRD files, and
logging to STDOUT for you to peruse in real time.

Typically there are two xymond_rrd processes running, one for status
messages and the other for data messages. The one we're interested in is
the one for the status messages, because the data are parsed from the
"disk" message body. You can run the following (as xymon user) to see
what's going on with this:

$ sudo -u xymon mkdir -p /tmp/myrrd
$ sudo -u xymon xymoncmd xymond_channel --channel=status \
 --filter='^@@status.*/hostname.example.com.*\|disk\|' xymond_rrd \
 --rrddir=/tmp/myrrd --debug

This will create RRD files in /tmp/myrrd for the named host and for only
the "disk" data. You can then use rrdtool to view the data and look for
signs of gaps. Note that xymond_rrd will cache updates to the RRD file, and
so you might not get updates to the RRD file straight away, so you can
either wait 30 minutes to review, or add the --no-cache switch. Also note
that RRD requires more than just one sample for it to start trusting the
data, so you would want to wait for at least two samples before relying on
the contents of the RRD file.

If that doesn't show any hints, then a more systematic approach may be
warranted. There are several stages between the Xymon client creating the
client data message and the RRD file being created, where messages could go
missing. Here's a rough summary:

xymon-client.sh => df
  => xymon [Client message]
  => xymond
  => xymond_client -> [status message]
  => xymond
  => xymond_channel --status
  => xymond_rrd
  => rrd

What I'd be doing is checking each phase of this to determine where the
messages are being dropped/corrupted. In most cases, you can impersonate
the existing process and review the results to infer what's happening in
the actual process.

J


On 10 August 2017 at 01:41, Stephen Barrie <sbarrie at brightsolid.com> wrote:

> Hello
>
>
>
> Attached are sample memory and disk graphs. You will see the gaps in one
> line do not equate with the gaps in others. The gaps vary in length and
> there is no clear pattern. Sometimes it is for minutes but they could be
> for more than an hour. The output from the df command in the client still
> updates the server every few minutes and this never seems to miss any data.
> It is difficult to get the df output that coincides with gaps in the graphs
> as we don’t see the graphs are missing data until after the fact, but we
> can see that the gaps are not at the same times for all machines or all
> disks.
>
>
>
> The server where we are not experiencing the problem is running 4.3.17 on
> Red Hat Linux. The others are 4.3.25 on Ubuntu 16.04 and 4.3.7 on Ubuntu
> 14.04. There are no RRD or other related errors in the logs.
>
>
>
> A problem with the RRD parser seems a logical explanation. Is there a way
> of tracking what this is doing?
>
>
>
> Stephen
>
>
>
> *From:* Jeremy Laidman [mailto:jlaidman at rebel-it.com.au]
> *Sent:* 09 August 2017 14:51
> *To:* Stephen Barrie <sbarrie at brightsolid.com>
> *Cc:* xymon at xymon.com
> *Subject:* Re: [Xymon] Graph lines for disk and memory showing gaps
>
>
>
> Can you show an example/screencap of a graph with the gaps? Do all servers
> and/or disks have gaps at the same time? How big are the gaps - one or two
> 5-minute sample size, or much larger?
>
>
>
> The Windows disk usage message is parsed by different code to that which
> parses the UNIX and Linux "df" output. That might have something to do with
> it.
>
>
>
> Is 4.3.25 in use on all Xymon servers?
>
>
>
> Check the Xymon logs for error messages. Perhaps the RRD parser is
> crashing.
>
>
>
> It's possible that the "df" output is sometimes not parseable by the RRD
> parser, and so the data is ignored. Can you show the [df] section from the
> client data during one of the gaps, and at another time when there are no
> gaps?
>
>
>
> J
>
>
>
>
>
> On 9 August 2017 at 22:13, Stephen Barrie <sbarrie at brightsolid.com> wrote:
>
> Hi
>
>
>
> We have a few Xymon servers running on Ubuntu 14.04 and 16.04 that show
> gaps in the graphs for disk and memory metrics. This specifically relates
> to Linux and Unix clients reporting to these servers. The graph lines for
> Windows clients look ok.
>
>
>
> If the same Unix and Linux clients are pointed to another server running
> Red Hat we do not see the same gaps in the graphs. Upgrading the client
> version does not resolve this. So it seems there is a problem with graph
> data being created on the Xymon server on Ubuntu. We are using the package
> version 4.3.25. Is there a known issue with this?
>
>
>
> Regards
>
>
>
> Stephen Barrie
> ------------------------------
> This message is confidential and may contain privileged information. You
> should not disclose its contents to any other person. If you are not the
> intended recipient, please notify the sender named above immediately. It is
> expressly declared that this e-mail does not constitute nor form part of a
> contract or unilateral obligation. Opinions, conclusions and other
> information in this message that do not relate to the official business of
> brightsolid shall be understood as neither given nor endorsed by it.
> ------------------------------
>
>
> __________________________________________________________________________
>
> This email has been checked for virus and other malicious content prior to
> leaving our network.
> __________________________________________________________________________
>
>
> _______________________________________________
> Xymon mailing list
> Xymon at xymon.com
> http://lists.xymon.com/mailman/listinfo/xymon
>
>
> ------------------------------
> This message is confidential and may contain privileged information. You
> should not disclose its contents to any other person. If you are not the
> intended recipient, please notify the sender named above immediately. It is
> expressly declared that this e-mail does not constitute nor form part of a
> contract or unilateral obligation. Opinions, conclusions and other
> information in this message that do not relate to the official business of
> brightsolid shall be understood as neither given nor endorsed by it.
> ------------------------------
>
> __________________________________________________________________________
>
> This email has been checked for virus and other malicious content prior to
> leaving our network.
> __________________________________________________________________________
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xymon.com/pipermail/xymon/attachments/20170810/c6b35c57/attachment.html>