[Xymon] Can custom client-side tests leverage the server-side value in analysis.cfg?

[Jeremy Laidman]

Grant, a few quick responses to your message:

The "DS" means "data source". It's a term used by the RRD (round-robin
database) library (and tools) that Xymon uses for storing historical data.

As a general principle, Xymon clients operate in local mode or central
mode. In (the older) local mode, the clients do all of their own analysis
and create their own data and status messages. In (the more modern) central
mode, the clients do a lot less of that, and instead simply gather data for
the server to parse and alert on.

For your requirement, what I would do is to have the client collect the
queue size (and other data) and include this in the client message and one
of the sections, eg [mailq]. The contents of this could simply be the
output of the "mailq" command. You could create a script in the "sections"
subdirectory, and the Xymon client will automatically run it and include
the output in the [mailq] section of the client message. Then on the
server, run a script (eg from tasks.cfg or something like tasks.d/mailq)
that dumps the client data message for every client, and if a [mailq]
section is found, create a status message.

And optionally turn the mail queue size into a graphable data point (either
create a trends message with the number, or use a colon-separated value
format for the Xymon parser to detect). The Xymon doco has a "how to
setup custom graphs" page with a few different ways to do this type of
thing.

J

On Sat, 14 Oct 2023 at 08:20, Grant Taylor via Xymon <xymon at xymon.com>
wrote:

>
>
>
> ---------- Forwarded message ----------
> From: Grant Taylor <gtaylor at tnetconsulting.net>
> To: xymon at xymon.com
> Cc:
> Bcc:
> Date: Fri, 13 Oct 2023 16:18:08 -0500
> Subject: Re: [Xymon] Can custom client-side tests leverage the server-side
> value in analysis.cfg?
> Hi Conor,
>
> Thank you for your reply.
>
> The overall gist of what I've understood thus far is that it's
> /indirectly/ possible to have custom client-side scripts use server-side
> values by arranging for a copy of those values to be available (cached)
> on the client for local use.  --  Is that an accurate summary?
>
> On 10/13/23 1:20 PM, Conor McCarthy wrote:
> > Yes, a simple way is to use the "config" command, this lets you
> > download any .cfg (or arbitrary files, if ALLOWALLCONFIGFILES="TRUE"
> > but don't do that :) files from the server's etc/ directory.
> >
> > client/bin/xymon $XYMONSERVER "config analysis.cfg" > tmp/analysis.cfg
> >
> > If you use "include" or "directory" directives in the .cfg files
> > Xymon will send you the "flattened" expanded contents.
>
> I'm glad to know that the analysis.cfg will be flattened as we're making
> extensive use of includes.
>
> > You will need to parse that file to do anything useful though.
> >
> > Alternatively, fetching the hosts.cfg and using xymongrep
> >    client/bin/xymon $XYMONSERVER "config hosts.cfg" > tmp/hosts.cfg
> >    HOSTSCFG=tmp/hosts.cfg client/bin/xymongrep mytag:*
> > might be simpler for simple configuration data that you can put in a
> > "tag:" for each host.
>
> I'd rather have the values being checked against -- as opposed to what
> tests are run -- in the analysis.cfg file.  It seems to me like having
> value entries in an additional file could potentially end up in some
> confusion.  Which file do I check CPU load in?  What about mail queue
> size?  Type thing.
>
> > xymongrep is normally used server-side to loop over multiple hosts,
> > you will likely need to match just one client name from its output
> > for a client script.
>
> That's the impression I had gotten.
>
> I feel like the ability to access the config, and thus what other
> clients are being monitored, is a little bit of an information leak.
> Probably not one that matters.  And it's not like I can do anything
> about that.
>
> > Alternatively, instead of using analysis.cfg the client-local.cfg
> > file is logically cleaner (and automatically shipped to clients, see
> > client/tmp/logfetch.HOSTNAME.cfg).
>
> ACK
>
> > This specifies the client-side part of log and file checks, although
> > it's not documented, you can add arbitrary lines to this file, e.g.
> >   [myhostname]
> >   x-my-specialvarA one
> >   x-my-specialvarB two
> >   log:/var/log/message:16384
> >   ignore MARK
>
> Good to know.
>
> > Each client will pick this up and store its own "personal" stanza
> > from the global client-local.cfg, this is automatically refreshed
> > each client run cycle.
>
> Does the server send the personalized version?  Or does the client
> receive the full version and only store it's own personal values?
>
> > Any non-documented parameters are explicitly ignored, but best placed
> > above any of the documented ones to make sure.
>
> Noted.
>
> > status: yes (though "clear" might be an option),
>
> Interesting call on the "clear" color.
>
> That has the added benefit of not declaring a green / yellow / read.
>
> > data: no.
>
> Noted.
>
> > There's also a "modify" message you can use after the fact to
> > change a status/colour (though take care not to trigger flapping).
>
> Ya.  I had seen modify.  I figured that might be part of how the server
> side tweaking of client side colors might happen.
>
> I believe there is some restriction on once a message has been modified,
> only the same thing can modify it again.  --  I assume that restriction
> is only there until a new status message comes in from the client.
>
> > Seems like fetching the hosts.cfg and using xymongrep should work
> > for that without too much trouble, at least as an exercise, and is
> > reasonably self-evident.
>
> ACK
>
> > hosts.cfg:
> >   10.1.2.3 mailhost1 #  ssh smtps MAILQ:qwarn=200,qcrit=500
> >
> > Then in a bash script (light on error checking) for example:
> >    client/bin/xymon $XYMONSERVER "fetch hosts.cfg" > tmp/hosts.cfg
> >    HOSTSCFG=tmp/hosts.cfg client/bin/xymongrep "MAILQ:*" |
> >    while read hostip hostname hostcfg; do
> >      if [[ "$hostname" == ${MACHINEDOTS} ]]; then
> >        mailcfg=""
> >        [[ "$hostcfg" =~ MAILQ:(.*) ]] && mailcfg="${BASH_REMATCH[1]}"
> >        for var in qwarn qcrit; do
> >          [[ "$mailcfg" =~ "($var)=([^,;]+) ]] && declare
> > "cfg_${BASH_REMATCH[1]}"="${BASH_REMATCH[2]}"
> >        done
> >        # declare -p ${!cfg_*} ##DEBUG##
> >        # mailq logic using $cfg_qwarn $cfg_qcrit goes here
> >        # xymon status submit goes here
> >      fi
> >    done
>
> This is some very tight and nice Bash.  I like how you're using test
> ([[) and parameter expansion.  --  Much to come back to and look at in
> more detail later.
>
> Aside:  Did you mean "fetch" or "config" when calling xymon?
>
> I think the biggest gotcha that I have is that some of the systems in my
> config are using different host names and relying on CLIENT: to make
> things match.  But I think that's a matter of finding and grinding out
> the CLIENT: from the ${hostcfg}.
>
> > You could dispense with the outer while loop by using
> >    read hostip hostname hostcfg < <(xymongrep MAILQ:* | grep
> "$MACHINEDOTS")
>
> ACK
>
> > There is no xymongrep equivalent for client-local.cfg, you could adapt
> > the above to parse the client cached copy (tmp/logfetch.HOSTNAME.cfg) -
> > simple enough since the server does not provide the entire client-local
> > config, just the client-specific matching stanza.
>
> I like the idea that the server only provides the client specific stanza.
>
> But I also like the idea that I can pull the analysis.cfg file using
> `${XYMON} ${XYMSRV} "config analysis.cfg"`.
>
> Initial skim of the xymongrep man page make me think that it's intended
> for the hosts.cfg file and not going to work with other files.  So I'd
> need to parse it.
>
> > I know there were some old, old mailq client scripts, there's still
> > some RRD related residue for parsing their output in Xymon. I've
> > never used it, but likely this was one:
> >
> https://web.archive.org/web/20080718210020/http://www.deadcat.net/viewfile.php?fileid=82
>
> $RESEARCH++
>
> > The bf/nmailq-bf.sh script therein simply barfs out a bit of sendmail's
> > "mailq" output. The point being if you have the data in an RRD then
> > you can also use "DS" in analysis.cfg to alert based on a GAUGE
> > type metric.  There's more than one way to skin a (dead)cat :)
>
> I have seen many references to "DS", but I'm still not quite sure what
> that means.  I need to read the custom graphing again and work to
> understand it.
>
> > To scale up the configuration complexity or number of discrete
> > service checks it would be better to dispense with xymongrep and
> > distribute your own easy-to-parse configuration file(s), or use the
> > client-local.cfg approach (my preference).
>
> ACK
>
> The ability to pull config files from the server makes this a bit
> easier.  It means that the client can get access to the configuration
> data to be able to more accurately set clear / green / yellow / red
> colors in reports.
>
> Thank you Conor.  I'll re-read, research, and do some thinking.
>
>
>
> --
> Grant. . . .
> unix || die
>
>
>
>
> ---------- Forwarded message ----------
> From: Grant Taylor via Xymon <xymon at xymon.com>
> To: xymon at xymon.com
> Cc:
> Bcc:
> Date: Fri, 13 Oct 2023 16:18:08 -0500
> Subject: Re: [Xymon] Can custom client-side tests leverage the server-side
> value in analysis.cfg?
> _______________________________________________
> Xymon mailing list
> Xymon at xymon.com
> http://lists.xymon.com/mailman/listinfo/xymon
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xymon.com/pipermail/xymon/attachments/20231015/d3099b73/attachment.htm>