[Xymon] [External] Xymon WinPSclient performance

Wed Jan 30 17:29:47 CET 2019

I agree with your observations and proposal.

Kind regards,

SebA

On Wed, 30 Jan 2019 at 15:59, Beck, Zak <zak.beck at accenture.com> wrote:

> The UTF8 / ASCII thing – interestingly, going right back to v1, the client
> has used an ASCII encoder. It has to use something because Socket.Send()
> expects a byte array, the data packet is constructed as a string and so you
> need to use an encoder from System.Text to convert the string to a byte
> array.
>
>
>
> The change in v2.21 was to add the Remove-Diacritics function, which is to
> strip out non-ASCII characters and also the 0xa0 non-breaking space. I can
> see how Remove-Diacritics might be a performance bottleneck on a long
> string.
>
>
>
> People with diacritics <https://en.wikipedia.org/wiki/Diacritic> may be
> using UTF8 anyway, I suspect most people using ASCII have no idea about the
> diacritic removal and have no need for it. Therefore I propose three values
> for the XymonAcceptUTF8 setting and this should impact on the minimum of
> people:
>
>
>
> 0 (the default) – use “original” ASCII, i.e. as per the original
> behaviour. This may cause diacritics to come through as unreadable
> multi-byte characters.
>
> 1 – use UTF8
>
> 2 – use “pure” ASCII, i.e. filter through Remove-Diacritics
>
>
>
> Zak
>
>
>
> *From:* Beck, Zak
> *Sent:* Wednesday, 30 January 2019 15:04
> *To:* 'SebA' <spah at syntec.co.uk>
> *Cc:* Xymon Mailing List <xymon at xymon.com>
> *Subject:* RE: [External] [Xymon] Xymon WinPSclient performance
>
>
>
> Yes, I remember our conversation – very difficult for me to replicate.
>
>
>
> I can definitely make this optional, and will most likely look to make
> this default to off.
>
>
>
> Zak
>
> *From:* SebA <spah at syntec.co.uk>
> *Sent:* Wednesday, 30 January 2019 14:57
> *To:* Beck, Zak <zak.beck at accenture.com>
> *Cc:* Xymon Mailing List <xymon at xymon.com>
> *Subject:* Re: [External] [Xymon] Xymon WinPSclient performance
>
>
>
> Good news and bad news.  The good news is that enabling XymonAcceptUTF8
> makes the cycle run a lot faster and seems to work fine for us, without
> doubling the number of bytes sent in the report.  I think the default
> though should probably be not to change the behaviour compared with the
> previous version (v2.19). Failing that, the default should probably be to
> enable XymonAcceptUTF8 - the default should not be something that slows
> down the cycle and uses lots of CPU time IMHO.
>
> The bad news is I have found another thing to takes time: XymonDiskPart.
>
> 2019-01-30 13:39:13  XymonDiskPart start
> 2019-01-30 13:39:17  XymonDiskPart finished
>
> I see this only runs every 'slow scan', but:
> (a) this also takes 4 seconds to complete;
> (b) I don't know what the use of it is, at least for most people, and it's
> not documented in XymonPSClient.doc - this is a possible example of feature
> bloat;
> (c) it causes a bad memory leak in the Virtual Disk service (vds.exe) on
> our failover cluster servers that have clustered disk.
>
> Commenting out line 3875 in v2.28 r8017 has no negative consequences for
> us and removes this vds.exe memory leak. I have compared patched and
> unpatched processes running for several months on otherwise identical
> servers. We have had a conversation about this, see here:
> https://lists.xymon.com/archive/2018-March/045341.html
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.xymon.com_archive_2018-2DMarch_045341.html&d=DwMFaQ&c=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU&r=S-aLwpx-PHBTBMIG_c2JczRC0SfuZCmsiH9Iams25FI&m=Yihzb9N4fl-Dl1BgOxMCrJsBMmLNdwI2ItiW5rRVcV8&s=-52Qhrm1-bvmWiTrhmSxyNmQdwM04ZP_KbPkNkWSIiA&e=>
> But you didn't respond to that last message. Since the Virtual Disk
> service was last restarted on the unpatched server on 30/11/18 (so in 2
> months), it has leaked up to today nearly 900 MB RAM and 9 GB of virtual
> memory.
>
> Please can XymonDiskPart be made optional in the config?
>
> We have another server (2012 R2, no failover cluster, no clustered disks)
> that is using 1 GB RAM and 2 GB virtual memory in the Xymon powershell
> process also (unpatched) v2.28 after 43 hours running. I haven't yet
> investigated the cause, but I'll try replacing it with the patched version
> without XymonDiskPart and see if that cures the memory leak.
>
> Kind regards,
>
> SebA
>
>
>
>
>
> On Tue, 29 Jan 2019 at 15:28, SebA <spah at syntec.co.uk> wrote:
>
> Hi Zak,
>
>
>
> Thanks, yes, haha, I did notice the irony myself and was going to point it
> out, or change my first paragraph to put the emphasis on reducing CPU time
> spent, but I forgot!  I noticed that powershell (on some versions of the
> PSclient) was using more CPU time than the process the server exists to
> run, which is clearly not a good situation.
>
>
>
> So if I select UTF-8, it's not going to double the size of the messages
> sent to the Xymon server?  I'll give it a go and see what happens.
>
>
>
> I think being able to configure the scan interval for CPUs would be good.
>
>
>
> The latest version of PSclient was not even completing one cycle before I
> was giving up and killing it, but I'll try UTF-8 and/or another server.
>
>
>
> Kind regards,
>
> SebA
>
>
>
>
>
> On Tue, 29 Jan 2019 at 11:59, Beck, Zak <zak.beck at accenture.com> wrote:
>
> Hi
>
>
>
> Yes, there is no option for no encoding – I think the closest option to
> this is actually UTF-8 because it does not attempt to adjust the message
> body (remove diacritics and 0xa0 spaces) before sending. That’s what is
> happening between your two log messages. Personally I don’t see that
> behaviour albeit with a 10x smaller data packet (I checked on several
> servers):
>
>
>
> 2019-01-29 11:43:06  Using ASCII encoding
>
> 2019-01-29 11:43:06  Connecting to host x.x.x.x
>
> 2019-01-29 11:43:06  Sent 100537 bytes to server
>
>
>
> We can add a third option to do no encoding I guess.
>
>
>
> Detecting CPUs is entirely down to this WMI call: Get-WmiObject -Class
> Win32_Processor. Yes, it does take time. I have tried over the years to
> reduce the amount of WMI calls (I think we’re down to 4) because they do
> seem to take a lot of time and on occasion are unreliable. Unfortunately I
> have not been able to find an alternative to this call. And I have 1000 VMs
> which can have CPUs hot added without reboot, so there is use case for
> checking every time. The result of the WMI call is already cached so we
> could amend the script to optionally run only on slow scans (every 6 hours
> by default).
>
>
>
> It may be simpler to add regex processing for your filename issue, I can
> take a look.
>
>
>
> I had to chuckle, I hope you appreciate the irony of saying we’re
> suffering from feature bloat and then requesting more enhancements /
> features 😉
>
>
>
> Zak
>
>
>
> *From:* Xymon <xymon-bounces at xymon.com> *On Behalf Of *SebA
> *Sent:* Monday, 28 January 2019 20:16
> *To:* Xymon Mailing List <xymon at xymon.com>
> *Subject:* [External] [Xymon] Xymon WinPSclient performance
>
>
>
> It's great the way features have been added to Xymon WinPSclient, but I
> think it's started suffering from feature bloat now, illustrating the
> importance of making each feature optional like BBWin did very effectively.
> We are seeing memory leaks on Windows 2012 R2 using version 2.28 (I started
> another thread about memory leaks a while back - I'll come back to this
> issue another time) and more and more CPU time being used up during each
> cycle.
>
> It seems that from version v2.21 a slow ASCII encoding conversion was
> added:
>
> myServer1 - : xymonclient.ps1  2.21 2017-04-28 zak.beck at accenture.com
> 2019-01-28 19:19:26  Sending to server
> 2019-01-28 19:19:26  Using ASCII encoding
> 2019-01-28 19:*20*:25  Connecting to host x.x.x.x        <-- Yes, that
> took about *1 minute of 100% processing of 1 of the 8 CPUs/Cores*.
> 2019-01-28 19:20:26  Sent 2007850 bytes to server    <-- OK, yes, that's
> rather a lot of data, but still it's not good.
> 2019-01-28 19:20:26  Received 130 bytes from server
>
> Compare with:
> myServer1 - : xymonclient.ps1  2.19 2016-12-28 zak.beck at accenture.com
> 2019-01-28 19:15:59  Sending to server
> 2019-01-28 19:15:59  Connecting to host x.x.x.x
> 2019-01-28 19:15:59  Sent 2006765 bytes to server
> 2019-01-28 19:16:00  Received 130 bytes from server
>
> Is it possible that converting to ASCII step introduced in v2.21 be made
> optional? In the latest version it is possible to send via UTF8, which was
> new in v2.21, or convert to ASCII, but it's not possible via configuration
> alone, I believe, to have the old behaviour (from v2.19) that worked fine
> for us and was far faster.
>
> Another slow process is detecting the number of CPUs (on the same server):
>
> 2019-01-28 19:29:05  XymonCollectInfo: Process info
> 2019-01-28 19:29:05  XymonCollectInfo: calling XymonProcsCPUUtilisation
> 2019-01-28 19:29:05  XymonCollectInfo: CPU info (WMI)
> 2019-01-28 19:29:13  Found 8 CPUs, total of 0 cores
> 2019-01-28 19:29:13  XymonCollectInfo: OS info (including memory) (WMI)
> 2019-01-28 19:29:13  XymonCollectInfo: Service info (WMI)
>
> *8 seconds* for that.  Why does it need to do this every time?  The
> number of CPUs does not normally change without at least rebooting.  It
> could do it just on the first cycle and cache it for future cycles.
>
> Another thing that would really speed up processing for us is being able
> to dynamically specify the filenames using special characters or something
> for the date, e.g. &DD &MM &YY
> We have files that rotate with new filesnames and if we could narrow down
> the filenames to process it would speed up processing by several more
> seconds.
> E.g. C:\Temp\myServerLog29-12-18.0.log
>
> In client-local.cfg:
>
> Instead of:
> log:C:\Temp\myServerLog*.log:10240
> If we could to this, it would mean we process a small fraction of the
> number of files:
> log:C:\Temp\myServerLog&DD-&MM-&YY.*.log:10240
>
>
> Kind regards,
>
> SebA
>
>
> ------------------------------
>
>
> This message is for the designated recipient only and may contain
> privileged, proprietary, or otherwise confidential information. If you have
> received it in error, please notify the sender immediately and delete the
> original. Any other use of the e-mail by you is prohibited. Where allowed
> by local law, electronic communications with Accenture and its affiliates,
> including e-mail and instant messaging (including content), may be scanned
> by our systems for the purposes of information security and assessment of
> internal compliance with Accenture policy. Your privacy is important to us.
> Accenture uses your personal data only in compliance with data protection
> laws. For further information on how Accenture processes your personal
> data, please see our privacy statement at
> https://www.accenture.com/us-en/privacy-policy.
>
> ______________________________________________________________________________________
>
> www.accenture.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xymon.com/pipermail/xymon/attachments/20190130/745b8f4a/attachment.html>