[Xymon] Formatting errors on log files

[Japheth Cleaver]

On 10/24/2016 10:59 AM, Greg Krpan wrote:
> I haven't noticed any errors through xymond_client or from debug mode.
>
> After running the xymoncmd line above I get the following:
>
> ./xymoncmd xymon 127.0.0.1 "xymondlog wstc-lr0dhcp1.svcs"
> wstc-lr0dhcp1|svcs|red||1477331698|1477331698|1477333498|0|0|15.1.160.11|460159|||Y|
> red Mon Oct 24 11:55:55 2016 - Services NOT ok
> &red BBWin: No matching service - want started/automatic
> &green DHCPServer is started/automatic - want started/automatic
> &green McAfeeFramework is started/automatic - want started/automatic
> &green McShield is started/automatic - want started/automatic
> &green McTaskManager is started/automatic - want started/automatic
> &green VMTools is started/automatic - want started/automatic
>
> Name                                StartupType  Status       DisplayName
> AeLookupSvc                         manual       stopped       
>  Application Experience
> ALG                                 manual       stopped       
>  Application Layer Gateway Service
> AppIDSvc                            manual       stopped       
>  Application Identity
> Appinfo                             manual
>   sta]ted        Application Information
> AppMgmt                             manual       stopped       
>  Application Management
> AppReadiness                        manual       stopped        App 
> Readiness
> AppXSvc                             manual       stopped        AppX 
> Deployment Service (AppXSVC)
> AudioEndpointBuilder                manual       stopped       
>  Windows Audio Endpoint Builder
> Audiosrv                  ]
>         manual       st]
> ped        Windows Audio
> ]BWin                               automatic    started        Big 
> Brother Xymon Client
> BFE                                 automatic    started        Base 
> Filtering Engine
*snip*

Thanks; that confirms that the issue involved xymond_client or xymond, 
and isn't related to the web display.

Looking through the changes from 4.3.25 to 4.3.27, it's hard to see what 
might be causing this issue.

Is there any chance you're under a significant memory pressure on this 
machine? Would you be able to add some glibc debugging at all?
If so, would you be able to add an:
(export) MALLOC_CHECK_=3
(export) MALLOC_PERTURB_=1
... into the environment? This might help trigger a memory issue that 
could otherwise go unnoticed.

Alternatively, the next step might be to downgrade to 4.3.25 and see if 
that fixes the problem (if so, that really indicated there's a specific 
hidden issue here). Also, it might be interesting to see if the el7 
Terabithia RPMs show the same problem for you. There was a significant 
increase in lookup/buffer debugging in xymond_client in there that's 
also in the 4.x-master branch but isn't in 4.3.x when compiled from source.

Regards,
-jc