[Xymon] Couple of questions on client data

[J.C. Cleaver]

On Sun, January 10, 2016 6:12 pm, David Boyer wrote:
> JC,
>       I see it takes stdout just fine.  Might I add that it might be
> beneficial if the script would output the heading
> say [script] vice the name of the the script "script.sh"?

True, but then you could just rename the script to "script" and leave off
the ".sh" :) As long as it's chmod +x (executable) and not named README or
a .rpm/.dpkg debris file it will be executed and added in.


> I java a bunch
> of specific info on each host that is 90%
> there for ref by our security folks, developers, etc... for instance which
> mode selinux is in, whether auditing is running,
> backup logs are current, etc.  Using the "local" directory feature would
> gather the info, put a header on it and allow
> reports to more easily be run against the data.

I can definitely see where you're coming from there, but the current
/local (and /sections, for Terabithia RPM users) functionality is geared
towards xymonclient.sh handling the wrapping of text with the [script]
headers itself.

Fortunately (or not), though, it's all just plain text once it gets to
xymond :) You could easily have your script "myscript" add in the headers
for sub-sections however you like...

===
#!/bin/sh

echo "this line get put into the xymonclient.sh-created [myscript] section"
echo "[foobar]"
/some/command/here --debug
cat /proc/some/other/data

echo "[snafu]"
/some/other/commands


=====
etc... It'll look and be treated just like any other [section].


Alternatively, you can have your script execute completely independently
of the default "xymonclient.sh" script. All you need to do is copy the
output format that xymonclient is generating, but use a custom
"collectorid" string:

  client[/COLLECTORID] HOSTNAME.OSTYPE [HOSTCLASS]

and then run it on a regular basis on the host. (This is convenient if you
have complex additional scripts or need to run something as root to
collect the data.) All of the un-expired client data will be concatenated
together when displayed/saved. If it can run as the xymon user, just make
a new section in the clientlaunch.cfg file on the client, and xymonlaunch
will handle it for you. It's up to you.


>
>       Also, on the xymond_rootlogin.pl script, where would this be set up
> to run? In the tasks.cfg?

Correct. This is server-side and should work pretty much as a drop-in
task. Take a look at how "xymond_client" is being run from there via
xymond_channel and duplicate that section (changing the name of the
header).

(There are multiple ways to do this sort of thing, especially at scale,
but this can get you started.)


-jc




> On Sun, Jan 10, 2016 at 8:05 PM, J.C. Cleaver <cleaver at terabithia.org>
> wrote:
>
>>
>> Glad it could help :)
>>
>> There should be a README-local file in the client home directory (or
>> documentation directory, depending on what install method you used). By
>> default, the client will look in $XYMONHOME/local/ for anything
>> executable. If anything is found, it's run (xymonclient.sh:65) and the
>> output is added to the client report near the end, under the heading
>> "[local:<filename>]". This allows a server admin to easily add new
>> things
>> on a one-off basis.
>>
>> Xymon v4.4 will have a similar directory named /sections/ with does the
>> same thing, except without the "local:" prefix on the output. (It's
>> intended for packaged site or distro/vendor-specific scripts.)
>>
>>
>> Ryan: Yes, this is one area where it definitely needs more highlighting.
>> A
>> lot of users miss that this happens, but it's really an invaluable
>> resource for forensics.
>>
>> Remember, though, that this requires xymon to be running in
>> server-config
>> mode, not --local client mode. In --local client mode, this raw data
>> never
>> leaves the original box and so can't be saved.
>>
>>
>> Regards,
>> -jc
>>
>>
>> On Sun, January 10, 2016 3:51 pm, David Boyer wrote:
>> > JC,
>> >       Yes, it does! Thanks for the background.. I've written/modified
>> > several tests for my environment while we were still in the "big
>> brother"
>> > era.  I'm still
>> > learning the ins/outs of xymon.  I'm intrigued by the use of the
>> "local"
>> > directory on the clientside.  Where might I find more information
>> about
>> > using this feature.
>> >
>> > Dave
>> >
>> > On Sun, Jan 10, 2016 at 3:35 PM, Novosielski, Ryan
>> > <novosirj at ca.rutgers.edu>
>> > wrote:
>> >
>> >> I just wanted to thank you, JC, for this information. I wanted to
>> know
>> >> whether it would be possible to check the kernel version back some
>> time
>> >> even though that is something I did not test. I figured the info
>> might
>> >> show
>> >> something, but that isn't kept in history or anything. If I had
>> realized
>> >> that a client data snapshot is kept when the status changes, I would
>> >> have
>> >> had the answer (which I still need, so it's very helpful).
>> >>
>> >> ____ *Note: UMDNJ is now Rutgers-Biomedical and Health Sciences*
>> >> || \\UTGERS      |---------------------*O*---------------------
>> >> ||_// Biomedical | Ryan Novosielski - Senior Technologist
>> >> || \\ and Health | novosirj at rutgers.edu- 973/972.0922 (2x0922)
>> >> ||  \\  Sciences | OIRT/High Perf & Res Comp - MSB C630, Newark
>> >>     `'
>> >>
>> >> On Jan 10, 2016, at 06:48, J.C. Cleaver <cleaver at terabithia.org>
>> wrote:
>> >>
>> >> Hi David,
>> >>
>> >>
>> >> On Sat, January 9, 2016 3:15 pm, David Boyer wrote:
>> >>
>> >> I see that there is data in the client data that is not turned into
>> >>
>> >> columns?  Can I turn this data into columns to report?  I'm using
>> xymon
>> >>
>> >> 4.3.20 on both the client/server.
>> >>
>> >>
>> >> On linux, I see that this data is not turned into any columns:
>> >>
>> >> [who]
>> >>
>> >> [route]
>> >>
>> >> [netstat]
>> >>
>> >> [ifstat]
>> >>
>> >>
>> >> Not that I'd use all of them, just wondering what the reasoning is
>> >> behind
>> >>
>> >> it.
>> >>
>> >>
>> >> This is correct. The client data is a full raw text of various bits
>> of
>> >> information that can be used to create status messages out of, but
>> there
>> >> isn't a mandated 1:1 correspondence between each section and each
>> test.
>> >> By
>> >> keeping them independent (raw data and central processing), you have
>> the
>> >> flexibility to write up new tests based off of already-existing
>> incoming
>> >> data, and/or add new "raw" data without having a specific test in
>> mind.
>> >>
>> >> For the 'who' data in particular, there's a sample processor in the
>> >> source
>> >> tarball that can demonstrate how easy it is to add simple new tests
>> at
>> >>
>> >>
>> https://sourceforge.net/p/xymon/code/HEAD/tree/branches/4.3.24/xymond/xymond_rootlogin.pl
>> >>
>> >> You can easily add new sections to the client data by adding files to
>> a
>> >> "/local/" directory on the client machine, or editing the
>> >> xymonclient-${OS}.sh shell script by hand, and running any command
>> that
>> >> the unprivileged 'xymon' user can execute.
>> >>
>> >> What's the benefit to all this additional data if it's not used?
>> >> Primarily
>> >> forensics and triage. As the xymon(7) man page puts it:
>> >>
>> >> The Xymon user-interface is simple, but engineers will also find lots
>> of
>> >>
>> >> relevant information. E.g. the data that clients report to Xymon
>> contain
>> >> the raw output from a number of system commands. That information is
>> >> available directly in Xymon, so an administrator no longer needs to
>> >> login to a server to get an overview of how it is behaving - the very
>> >> commands they would normally run have already been performed, and the
>> >> results are on-line in Xymon.
>> >>
>> >> https://www.xymon.com/help/manpages/man7/xymon.7.html
>> >>
>> >>
>> >>
>> >> This becomes even more relevant when you consider snapshoting. When a
>> >> status goes "red", a snapshot of the client data at that time is
>> kept.
>> >> So
>> >> if you went back later to try to figure out why (e.g.) CPU was
>> rising,
>> >> the
>> >> output of the '[who]' section tells you who might have been doing
>> >> something then, even if the data wasn't used for making a test out of
>> at
>> >> that time.
>> >>
>> >>
>> >>
>> >>
>> >> Also, I notice that getting the data via xymondboard is not 100%
>> >> either...
>> >>
>> >>
>> >> [xymon at ztest bin]$ ./xymon 192.168.1.230 "clientlog yumlist
>> section=who"
>> >>
>> >> *snip*
>> >>
>> >>
>> >> As you can see below, requesting a status on all the tests, there is
>> no
>> >>
>> >> "who" being reported.
>> >>
>> >> But just a moment ago, asking for just that data it works.
>> >>
>> >>
>> >> [xymon at ztest bin]$ ./xymon 192.168.1.230 "xymondboard host=yumlist"
>> >>
>> >> *snip*
>> >>
>> >>
>> >> Correct. The "clientlog" command retrieves the most recent raw client
>> >> data, while the "xymondboard" command retrieves just the status
>> messages
>> >> (the tests that you see on the webpages).
>> >>
>> >> The "clientlog" column on the web page, much like the "info" and
>> >> "trends"
>> >> columns, isn't a real test... it's just present to provide easy
>> access
>> >> to
>> >> the most recent data. You can also access it from the "Client data"
>> link
>> >> on the bottom of any of that host's status pages.
>> >>
>> >>
>> >> Hope that helps!
>> >>
>> >> -jc
>> >>
>> >> _______________________________________________
>> >> Xymon mailing list
>> >> Xymon at xymon.com
>> >> http://lists.xymon.com/mailman/listinfo/xymon
>> >>
>> >>
>> >
>>
>>
>>
>