[Xymon] Can custom client-side tests leverage the server-side value in analysis.cfg?

[Grant Taylor]

Hi Conor,

Thank you for your reply.

The overall gist of what I've understood thus far is that it's 
/indirectly/ possible to have custom client-side scripts use server-side 
values by arranging for a copy of those values to be available (cached) 
on the client for local use.  --  Is that an accurate summary?

On 10/13/23 1:20 PM, Conor McCarthy wrote:
> Yes, a simple way is to use the "config" command, this lets you 
> download any .cfg (or arbitrary files, if ALLOWALLCONFIGFILES="TRUE" 
> but don't do that :) files from the server's etc/ directory.
> 
> client/bin/xymon $XYMONSERVER "config analysis.cfg" > tmp/analysis.cfg
> 
> If you use "include" or "directory" directives in the .cfg files 
> Xymon will send you the "flattened" expanded contents.

I'm glad to know that the analysis.cfg will be flattened as we're making 
extensive use of includes.

> You will need to parse that file to do anything useful though.
> 
> Alternatively, fetching the hosts.cfg and using xymongrep
>    client/bin/xymon $XYMONSERVER "config hosts.cfg" > tmp/hosts.cfg
>    HOSTSCFG=tmp/hosts.cfg client/bin/xymongrep mytag:*
> might be simpler for simple configuration data that you can put in a 
> "tag:" for each host.

I'd rather have the values being checked against -- as opposed to what 
tests are run -- in the analysis.cfg file.  It seems to me like having 
value entries in an additional file could potentially end up in some 
confusion.  Which file do I check CPU load in?  What about mail queue 
size?  Type thing.

> xymongrep is normally used server-side to loop over multiple hosts, 
> you will likely need to match just one client name from its output 
> for a client script.

That's the impression I had gotten.

I feel like the ability to access the config, and thus what other 
clients are being monitored, is a little bit of an information leak. 
Probably not one that matters.  And it's not like I can do anything 
about that.

> Alternatively, instead of using analysis.cfg the client-local.cfg 
> file is logically cleaner (and automatically shipped to clients, see 
> client/tmp/logfetch.HOSTNAME.cfg).

ACK

> This specifies the client-side part of log and file checks, although 
> it's not documented, you can add arbitrary lines to this file, e.g.
>   [myhostname]
>   x-my-specialvarA one
>   x-my-specialvarB two
>   log:/var/log/message:16384
>   ignore MARK

Good to know.

> Each client will pick this up and store its own "personal" stanza 
> from the global client-local.cfg, this is automatically refreshed 
> each client run cycle.

Does the server send the personalized version?  Or does the client 
receive the full version and only store it's own personal values?

> Any non-documented parameters are explicitly ignored, but best placed 
> above any of the documented ones to make sure.

Noted.

> status: yes (though "clear" might be an option),

Interesting call on the "clear" color.

That has the added benefit of not declaring a green / yellow / read.

> data: no.

Noted.

> There's also a "modify" message you can use after the fact to 
> change a status/colour (though take care not to trigger flapping).

Ya.  I had seen modify.  I figured that might be part of how the server 
side tweaking of client side colors might happen.

I believe there is some restriction on once a message has been modified, 
only the same thing can modify it again.  --  I assume that restriction 
is only there until a new status message comes in from the client.

> Seems like fetching the hosts.cfg and using xymongrep should work 
> for that without too much trouble, at least as an exercise, and is 
> reasonably self-evident.

ACK

> hosts.cfg:
>   10.1.2.3 mailhost1 #  ssh smtps MAILQ:qwarn=200,qcrit=500
> 
> Then in a bash script (light on error checking) for example:
>    client/bin/xymon $XYMONSERVER "fetch hosts.cfg" > tmp/hosts.cfg
>    HOSTSCFG=tmp/hosts.cfg client/bin/xymongrep "MAILQ:*" |
>    while read hostip hostname hostcfg; do
>      if [[ "$hostname" == ${MACHINEDOTS} ]]; then
>        mailcfg=""
>        [[ "$hostcfg" =~ MAILQ:(.*) ]] && mailcfg="${BASH_REMATCH[1]}"
>        for var in qwarn qcrit; do
>          [[ "$mailcfg" =~ "($var)=([^,;]+) ]] && declare 
> "cfg_${BASH_REMATCH[1]}"="${BASH_REMATCH[2]}"
>        done
>        # declare -p ${!cfg_*} ##DEBUG##
>        # mailq logic using $cfg_qwarn $cfg_qcrit goes here
>        # xymon status submit goes here
>      fi
>    done

This is some very tight and nice Bash.  I like how you're using test 
([[) and parameter expansion.  --  Much to come back to and look at in 
more detail later.

Aside:  Did you mean "fetch" or "config" when calling xymon?

I think the biggest gotcha that I have is that some of the systems in my 
config are using different host names and relying on CLIENT: to make 
things match.  But I think that's a matter of finding and grinding out 
the CLIENT: from the ${hostcfg}.

> You could dispense with the outer while loop by using
>    read hostip hostname hostcfg < <(xymongrep MAILQ:* | grep "$MACHINEDOTS")

ACK

> There is no xymongrep equivalent for client-local.cfg, you could adapt
> the above to parse the client cached copy (tmp/logfetch.HOSTNAME.cfg) -
> simple enough since the server does not provide the entire client-local
> config, just the client-specific matching stanza.

I like the idea that the server only provides the client specific stanza.

But I also like the idea that I can pull the analysis.cfg file using 
`${XYMON} ${XYMSRV} "config analysis.cfg"`.

Initial skim of the xymongrep man page make me think that it's intended 
for the hosts.cfg file and not going to work with other files.  So I'd 
need to parse it.

> I know there were some old, old mailq client scripts, there's still 
> some RRD related residue for parsing their output in Xymon. I've 
> never used it, but likely this was one:
> https://web.archive.org/web/20080718210020/http://www.deadcat.net/viewfile.php?fileid=82

$RESEARCH++

> The bf/nmailq-bf.sh script therein simply barfs out a bit of sendmail's 
> "mailq" output. The point being if you have the data in an RRD then 
> you can also use "DS" in analysis.cfg to alert based on a GAUGE 
> type metric.  There's more than one way to skin a (dead)cat :)

I have seen many references to "DS", but I'm still not quite sure what 
that means.  I need to read the custom graphing again and work to 
understand it.

> To scale up the configuration complexity or number of discrete 
> service checks it would be better to dispense with xymongrep and 
> distribute your own easy-to-parse configuration file(s), or use the 
> client-local.cfg approach (my preference).

ACK

The ability to pull config files from the server makes this a bit 
easier.  It means that the client can get access to the configuration 
data to be able to more accurately set clear / green / yellow / red 
colors in reports.

Thank you Conor.  I'll re-read, research, and do some thinking.



-- 
Grant. . . .
unix || die