[Xymon] [External] Xymon WinPSclient performance

Wed Jan 30 11:57:03 CET 2019

Zak,

I forgot to mention:
regex may not cut it for the filename issue - it's placeholders for parts
of the date that would be useful here, where that placeholder is
substituted with that part of the date, e.g. in Python:
%d -- Day of the month as a zero-padded decimal number - ref:
http://strftime.org/
In .NET (PowerShell):
https://docs.microsoft.com/en-us/dotnet/standard/base-types/custom-date-and-time-format-strings
We just need a way of specifying the date format we want to use and it
could be interpreted by the

ToString

method of the date and time instance that holds the current datetime.

So, to add to what I had in my original post, in client-local.cfg:
Instead of:
log:C:\Temp\myServerLog*.log:10240
If we could to this, it would mean we process a small fraction of the
number of files:
log:C:\Temp\myServerLog&DD-&MM-&YY.*.log:10240
This would be interpreted at the client side as (when the date is
2019-01-30):
C:\Temp\myServerLog30-01-19.*.log
Which would find and process, e.g. C:\Temp\myServerLog30-01-19.0.log and
C:\Temp\myServerLog30-01-19.1.log

Kind regards,

SebA

On Tue, 29 Jan 2019 at 15:28, SebA <spah at syntec.co.uk> wrote:

> Hi Zak,
>
> Thanks, yes, haha, I did notice the irony myself and was going to point it
> out, or change my first paragraph to put the emphasis on reducing CPU time
> spent, but I forgot!  I noticed that powershell (on some versions of the
> PSclient) was using more CPU time than the process the server exists to
> run, which is clearly not a good situation.
>
> So if I select UTF-8, it's not going to double the size of the messages
> sent to the Xymon server?  I'll give it a go and see what happens.
>
> I think being able to configure the scan interval for CPUs would be good.
>
> The latest version of PSclient was not even completing one cycle before I
> was giving up and killing it, but I'll try UTF-8 and/or another server.
>
> Kind regards,
>
> SebA
>
>
>
> On Tue, 29 Jan 2019 at 11:59, Beck, Zak <zak.beck at accenture.com> wrote:
>
>> Hi
>>
>>
>>
>> Yes, there is no option for no encoding – I think the closest option to
>> this is actually UTF-8 because it does not attempt to adjust the message
>> body (remove diacritics and 0xa0 spaces) before sending. That’s what is
>> happening between your two log messages. Personally I don’t see that
>> behaviour albeit with a 10x smaller data packet (I checked on several
>> servers):
>>
>>
>>
>> 2019-01-29 11:43:06  Using ASCII encoding
>>
>> 2019-01-29 11:43:06  Connecting to host x.x.x.x
>>
>> 2019-01-29 11:43:06  Sent 100537 bytes to server
>>
>>
>>
>> We can add a third option to do no encoding I guess.
>>
>>
>>
>> Detecting CPUs is entirely down to this WMI call: Get-WmiObject -Class
>> Win32_Processor. Yes, it does take time. I have tried over the years to
>> reduce the amount of WMI calls (I think we’re down to 4) because they do
>> seem to take a lot of time and on occasion are unreliable. Unfortunately I
>> have not been able to find an alternative to this call. And I have 1000 VMs
>> which can have CPUs hot added without reboot, so there is use case for
>> checking every time. The result of the WMI call is already cached so we
>> could amend the script to optionally run only on slow scans (every 6 hours
>> by default).
>>
>>
>>
>> It may be simpler to add regex processing for your filename issue, I can
>> take a look.
>>
>>
>>
>> I had to chuckle, I hope you appreciate the irony of saying we’re
>> suffering from feature bloat and then requesting more enhancements /
>> features 😉
>>
>>
>>
>> Zak
>>
>>
>>
>> *From:* Xymon <xymon-bounces at xymon.com> *On Behalf Of *SebA
>> *Sent:* Monday, 28 January 2019 20:16
>> *To:* Xymon Mailing List <xymon at xymon.com>
>> *Subject:* [External] [Xymon] Xymon WinPSclient performance
>>
>>
>>
>> It's great the way features have been added to Xymon WinPSclient, but I
>> think it's started suffering from feature bloat now, illustrating the
>> importance of making each feature optional like BBWin did very effectively.
>> We are seeing memory leaks on Windows 2012 R2 using version 2.28 (I started
>> another thread about memory leaks a while back - I'll come back to this
>> issue another time) and more and more CPU time being used up during each
>> cycle.
>>
>> It seems that from version v2.21 a slow ASCII encoding conversion was
>> added:
>>
>> myServer1 - : xymonclient.ps1  2.21 2017-04-28 zak.beck at accenture.com
>> 2019-01-28 19:19:26  Sending to server
>> 2019-01-28 19:19:26  Using ASCII encoding
>> 2019-01-28 19:*20*:25  Connecting to host x.x.x.x        <-- Yes, that
>> took about *1 minute of 100% processing of 1 of the 8 CPUs/Cores*.
>> 2019-01-28 19:20:26  Sent 2007850 bytes to server    <-- OK, yes, that's
>> rather a lot of data, but still it's not good.
>> 2019-01-28 19:20:26  Received 130 bytes from server
>>
>> Compare with:
>> myServer1 - : xymonclient.ps1  2.19 2016-12-28 zak.beck at accenture.com
>> 2019-01-28 19:15:59  Sending to server
>> 2019-01-28 19:15:59  Connecting to host x.x.x.x
>> 2019-01-28 19:15:59  Sent 2006765 bytes to server
>> 2019-01-28 19:16:00  Received 130 bytes from server
>>
>> Is it possible that converting to ASCII step introduced in v2.21 be made
>> optional? In the latest version it is possible to send via UTF8, which was
>> new in v2.21, or convert to ASCII, but it's not possible via configuration
>> alone, I believe, to have the old behaviour (from v2.19) that worked fine
>> for us and was far faster.
>>
>> Another slow process is detecting the number of CPUs (on the same server):
>>
>> 2019-01-28 19:29:05  XymonCollectInfo: Process info
>> 2019-01-28 19:29:05  XymonCollectInfo: calling XymonProcsCPUUtilisation
>> 2019-01-28 19:29:05  XymonCollectInfo: CPU info (WMI)
>> 2019-01-28 19:29:13  Found 8 CPUs, total of 0 cores
>> 2019-01-28 19:29:13  XymonCollectInfo: OS info (including memory) (WMI)
>> 2019-01-28 19:29:13  XymonCollectInfo: Service info (WMI)
>>
>> *8 seconds* for that.  Why does it need to do this every time?  The
>> number of CPUs does not normally change without at least rebooting.  It
>> could do it just on the first cycle and cache it for future cycles.
>>
>> Another thing that would really speed up processing for us is being able
>> to dynamically specify the filenames using special characters or something
>> for the date, e.g. &DD &MM &YY
>> We have files that rotate with new filesnames and if we could narrow down
>> the filenames to process it would speed up processing by several more
>> seconds.
>> E.g. C:\Temp\myServerLog29-12-18.0.log
>>
>> In client-local.cfg:
>>
>> Instead of:
>> log:C:\Temp\myServerLog*.log:10240
>> If we could to this, it would mean we process a small fraction of the
>> number of files:
>> log:C:\Temp\myServerLog&DD-&MM-&YY.*.log:10240
>>
>>
>> Kind regards,
>>
>> SebA
>>
>> ------------------------------
>>
>> This message is for the designated recipient only and may contain
>> privileged, proprietary, or otherwise confidential information. If you have
>> received it in error, please notify the sender immediately and delete the
>> original. Any other use of the e-mail by you is prohibited. Where allowed
>> by local law, electronic communications with Accenture and its affiliates,
>> including e-mail and instant messaging (including content), may be scanned
>> by our systems for the purposes of information security and assessment of
>> internal compliance with Accenture policy. Your privacy is important to us.
>> Accenture uses your personal data only in compliance with data protection
>> laws. For further information on how Accenture processes your personal
>> data, please see our privacy statement at
>> https://www.accenture.com/us-en/privacy-policy.
>>
>> ______________________________________________________________________________________
>>
>> www.accenture.com
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xymon.com/pipermail/xymon/attachments/20190130/820734a5/attachment.html>