[Xymon] Xymon script for linux

[Japheth Cleaver]

On 9/23/2019 4:46 AM, Bruce Ferrell wrote:
> On 9/22/19 6:31 PM, Jeremy Laidman wrote:
>> On Sat, 21 Sep 2019 at 10:30, Bruce Ferrell <bferrell at baywinds.org 
>> <mailto:bferrell at baywinds.org>> wrote:
>>
>>
>>     I'm curious how you propose actually using sar data in a client 
>> message.  On my system /var/log/sa is 44M of text data.
>>
>>
>> Yeah, I could have been clearer in describing what I had in mind.
>>
>>
>>     vmstat (procps) is already used and feeds rrd on the server via 
>> the client message; which is a MUCH more compact, long term storage 
>> format (The rrds for the same system are
>>     only 13M).
>>
>>
>> If "sar [some options]" can produce equivalent output to (say) 
>> "vmstat 300 2", then it's likely to take the same amount of storage 
>> and bytes transmitted. But it has the added benefit that the same sar 
>> output from Solaris and AIX and Linux can all be parsed by the same 
>> code on the Xymon server. So the one client-side command to create 
>> [vmstat] in the client message will be universal across a wide range 
>> of OSes. If we can replace [vmstat] in this way, we may be able to do 
>> the same for [netstat] and [iostat] and a few others (by using sar 
>> with other options). We might also introduce an equivalent to 
>> [iostatdisk] and other things that don't yet exist in some OSes' 
>> client messages.
>>
> Interesting idea... but sa doesn't work like that at all.
>
> sar can collect individualized sets of stats, but the default the cron 
> jobs that come in the sysstat package run sa to collect all system 
> stats at "cron interval". Then it has to be sliced and diced after the 
> fact.... It's semi non-trivial to extract "just the latest update 
> changes for the stat set you want" to transmit the the server.  Not 
> that it can't be done, but it's a pain.
>
> SO "just drop sar on... " become "just drop sar and modify it's 
> defaults".  If sar is already there and I'm the admin, I'm going to be 
> a wee touch miffed if someone messes with my system wide, long term 
> stats collection or creates a duplicate, parallel, space gobbling 
> collection dataset at the client.

That's basically how the version I had spec'd out operated. It would be 
a set of parallel sadc processes writing out to temp files, get turned 
into textual output by 'sar', and then payloaded up in the client message:

 > $SADC -F ${SADCOPTS} -L ${SADCFREQ} 
$XYMONTMP/xymon_sadc.$MACHINEDOTS.$$; $SAR -A -f 
$XYMONTMP/xymon_sadc.$MACHINEDOTS.$$ > 
$XYMONTMP/xymon_sar.$MACHINEDOTS.$$; mv 
$XYMONTMP/xymon_sar.$MACHINEDOTS.$$ $XYMONTMP/xymon_sar.$MACHINEDOTS; rm 
-f $XYMONTMP/xymon_sadc.$MACHINEDOTS.$$

Another option would be to transmit just the sadc binary datafile, but 
it seemed likely that could vary in structure among differing releases, 
while sar-processing scripts and tools that can interpret the text feed 
are pretty numerous.

As a side benefit, having per-minute sar output for the 5 minutes prior 
to an event occurring in the client message could be useful for postmortems.

The downside, of course, is that the text output is indeed huge. Even if 
it's transmitted compressed, it's still greatly bumping up your resource 
utilization. That suggests that the benefits of having full text output 
in the client message might be outweighed by the space savings in just 
having clients send direct "sar" data messages back themselves.


>
> vm/io/whatever stat collects a sample and sends just that sample. All 
> storage is at the xymon server  MUCH cleaner even if adding the 
> rare/exotic new thing is harder when they come up.  BTW, sar on OS X 
> is a way different beast too, so it violates the premise that the same 
> code works.  but *stat works out of the box.

Of course OS X has to be different :)

-jc