[Xymon] Formatting errors on log files

J.C. Cleaver cleaver at terabithia.org
Sat Oct 22 01:24:47 CEST 2016


Hi,

On Fri, October 21, 2016 9:19 am, Greg Krpan wrote:
> As an additional update, I've checked multiple times and the raw data that
> is passed to the system is correct.  It appears to be when it is displayed
> on the webpage where the problem occurs. I have not updated xymon- it
> appears as though the version that I am on is still the most recent
> version
> (4.3.27)  Is anyone aware of any conflicts that may have been introduced
> after patching?  Since this is a production server, I run patches on a
> monthly basis, with the most recent patching occurring on 9/29, which is
> coincidentally when the problem started occurring.


Can you run from the command line:
xymoncmd xymon 127.0.0.1 "xymondlog <hostname>.<affectedtestname>"
for a test (like svcs) that you're seeing now? I'm curious if the garbled
data is showing in memory or if it's happening (just) on the web layer.

I'm not aware of any outstanding issues in xymond_client code that
wouldn't be affecting lots of people, but it's possible we have a bug
here. Most of the recent changes have been either new features or CSP/XSS
fixes at the display layer.


Have you noticed any errors coming through xymond_client, or any patterns
from running it in --debug mode?

-jc



>
> On Mon, Oct 17, 2016 at 4:27 PM, Greg Krpan <gjkrpan43 at gmail.com> wrote:
>
>> I've included an entire "SVCS" status below on a failed status screen.
>> As
>> you can see, it is random as to how and where the output corrupts,
>>
>> On the Windows systems, I run BBWin for the client (version 0.13).  I
>> should be able to put the xymond_client process into debug mode to
>> monitor
>> for a while as well..  The problem is less predominant on Linux clients
>> than on Windows, but it is occurring on both.
>>
>> The server is running on CentOS 7 with current patches.
>> # uname -a
>> Linux ************************* 3.10.0-327.36.1.el7.x86_64 #1 SMP Sun
>> Sep
>> 18 13:04:29 UTC 2016 x86_64 x86_64 x86_64 GNU/Linu
>>
>> # cat /etc/centos-release
>> CentOS Linux release 7.2.1511 (Core)
>>
>> [# cat /etc/os-release
>> NAME="CentOS Linux"
>> VERSION="7 (Core)"
>> ID="centos"
>> ID_LIKE="rhel fedora"
>> VERSION_ID="7"
>> PRETTY_NAME="CentOS Linux 7 (Core)"
>> ANSI_COLOR="0;31"
>> CPE_NAME="cpe:/o:centos:centos:7"
>> HOME_URL="https://www.centos.org/"
>> BUG_REPORT_URL="https://bugs.centos.org/"
>>
>> CENTOS_MANTISBT_PROJECT="CentOS-7"
>> CENTOS_MANTISBT_PROJECT_VERSION="7"
>> REDHAT_SUPPORT_PRODUCT="centos"
>> REDHAT_SUPPORT_PRODUCT_VERSION="7"
>>
>>
>> Name                                StartupType  Status
>> DisplayName
>> AeLookupSvc                         manual       stopped
>> Application Experience
>> ALG                                 manual       stopped
>> Application Layer Gateway Service
>> AppHostSvc                          automatic    started
>> Application Host Helper Service
>> AppIDSvc                            manual       stopped
>> Application Identity
>> Appinfo                             manual       stopped
>> Application I
>> forma]ion
>> AppMgmt                             manual       stopped
>> Application Management
>> AppReadiness                        manual       stopped        App
>> Readiness
>> AppXSvc                             manual       stopped        AppX
>> Deployment Service (AppXSVC)
>> aspnet_state                        manual       stopped        ASP.NET
>> State Service
>> AudioEndpointBuilder                man
>> al       stopped        Wi]dows Audio Endpoint Builder
>> Audiosrv                            manual       stopped        Windows
>> Audio
>> BBWin                               automatic    started        Big
>> Brother Xymon Client
>> BFE                                 automatic    started        Base
>> Filtering Engine
>> BITS                                automatic    started        Back
>> round Intelligent Trans]
>> r Service
>> BrokerInfrastru]
>> ure                automatic ]
>>  started        Background Tasks]
>> nfrastructure Service
>> Browser        ]
>>                    ]isabled     stopped        Computer Browser
>> CcmExec                             automatic    started        SMS
>> Agent Host
>> CertPropSvc                         manual       started
>> Certificate Propagation
>> CmRcService                         disabled     stopped
>> Configuration Manager Remote Control
>> COMSysApp                           manual       started        COM+
>> System Application
>> CryptSvc                            automatic    started
>> Cryptographic Services
>> DcomLaunch                          automatic    started        DCOM
>> Server Process Launcher
>> defragsvc                           manual       stopped        Optimize
>> drives
>> DeviceAssociationService            manual       stopped        Device
>> Association Service
>> DeviceInstall                       manual       stopped        Device
>> Install Service
>> Dhcp                                automatic    started        DHCP
>> Client
>> DiagTrack                           automatic    started
>> Diagnostics Tracking Service
>> Dnscache                            automatic    started        DNS
>> Client
>> dot3svc                             manual       stopped        Wired
>> AutoConfig
>> DPS                                 automatic    started
>> Diagnostic Policy Service
>> DsmSvc                              manual       started        Device
>> Setup Manager
>> Eaphost                             manual       stopped
>> Extensible Authentication Protocol
>> EFS                                 manual       stopped
>> Encrypting File System (EFS)
>> EventLog                            automatic    started        Windows
>> Event Log
>> EventSystem                         automatic    started        COM+
>> Event System
>> fdPHost                             manual       stopped        Function
>> Discovery Provider Host
>> FDResPub                            manual       stopped        Function
>> Discovery Resource Publication
>> FontCache                           automatic    started        Windows
>> Font Cache Service
>> gpsvc                               automatic    started        Group
>> Policy Client
>> hidserv                             manual       stopped        Human
>> Interface Device Service
>> hkmsvc                              manual       stopped        Health
>> Key and Certificate Management
>> IEEtwCollectorService               manual       stopped        Internet
>> Explorer ETW Collector Service
>> IISADMIN                            automatic    started        IIS
>> Admin Service
>> IKEEXT                              automatic    started        IKE and
>> AuthIP IPsec Keying Modules
>> iphlpsvc                            automatic    started        IP
>> Helper
>> KeyIso                              manual       started        CNG Key
>> Isolation
>> KPSSVC                              manual       stopped        KDC
>> Proxy Server service (KPS)
>> KtmRm                               manual       stopped        KtmRm
>> for Distributed Transaction Coordinator
>> LanmanServer                        automatic    started        Server
>> LanmanWorkstation                   automatic    started
>> Workstation
>> lltdsvc                             manual       stopped
>> Link-Layer Topology Discovery Mapper
>> lmhosts                             automatic    started        TCP/IP
>> NetBIOS Helper
>> lpasvc                              manual       stopped
>> Microsoft Policy Platform Local Authority
>> lppsvc                              manual       stopped
>> Microsoft Policy Platform Processor
>> LSM                                 automatic    started        Local
>> Session Manager
>> McAfeeFramework                     automatic    started        McAfee
>> Framework Service
>> McShield                            automatic    started        McAfee
>> McShield
>> McTaskManager                       automatic    started        McAfee
>> Task Manager
>> MMCSS                               manual       stopped
>> Multimedia Class Scheduler
>> MpsSvc
>>  automatic    started  ]     Windows Firewall
>> MSDTC                               automatic    started
>> Distributed Transaction Coordinator
>> MSiSCSI                             manual       stopped
>> Microsoft iSCSI Initiator Service
>> msiserver                           manual       stopped        Windows
>> Installer
>> napagent                            manual       stopped        Network
>> Access Protection Agent
>> NcaSvc                              manual       stopped        Network
>> Connectivity Assistant
>> Netlogon                            automatic    started        Netlogon
>> Netman                              manual       stopped        Network
>> Connections
>> netprofm                            manual       started        Network
>> List Service
>> NetTcpPortSharing                   disabled     stopped        Net.Tcp
>> Port Sharing Service
>> NlaSvc                              automatic    started        Network
>> Location Awareness
>> nsi                                 automatic    started        Network
>> Store Interface Service
>> PerfHost                            manual       stopped
>> Performance Counter DLL Host
>> pla                                 manual       stopped
>> Performance Logs & Alerts
>> PlugPlay                            manual       started        Plug and
>> Play
>> PolicyAgent                         manual       started        IPsec
>> Policy Agent
>> Power                               automatic    started        Power
>> PrintNotify                         manual       stopped        Printer
>> Extensions and Notifications
>> ProfSvc                             automatic    started        User
>> Profile Service
>> QBCFMonitorService                  automatic    started
>> QuickBooks Database Manager Service
>> QBFCService                         manual       stopped        Intuit
>> QuickBooks FCS
>> QuickBooksDB17                      automatic    started
>> QuickBooksDB17
>> RasAuto                             manual       stopped        Remote
>> Access Auto Connection Manager
>> RasMan                              manual       stopped        Remote
>> Access Connection Manager
>> RemoteAccess                        disabled     stopped        Routing
>> and Remote Access
>> RemoteRegistry                      automatic    stopped        Remote
>> Registry
>> RpcEptMapper                        automatic    started        RPC
>> Endpoint Mapper
>> RpcLocator                          manual       stopped        Remote
>> Procedure Call (RPC) Locator
>> RpcSs                               automatic    started        Remote
>> Procedure Call (RPC)
>> RSoPProv                            manual       stopped
>> Resultant Set of Policy Provider
>> sacsvr                              manual       stopped        Special
>> Administration Console Helper
>> SamSs                               automatic    started        Security
>> Accounts Manager
>> SCardSvr                            disabled     stopped        Smart
>> Card
>> ScDeviceEnum                        manual       stopped        Smart
>> Card Device Enumeration Service
>> Schedule                            automatic    started        Task
>> Scheduler
>> SCPolicySvc                         manual       stopped        Smart
>> Card Removal Policy
>> seclogon                            manual       stopped
>> Secondary Logon
>> SENS                                automatic    started        System
>> Event Notification Service
>> SessionEnv                          manual       started        Remote
>> Desktop Configuration
>> SharedAccess                        disabled     stopped        Internet
>> Connection Sharing (ICS)
>> ShellHWDetection                    automatic    stopped        Shell
>> Hardware Detection
>> smphost                             manual       stopped
>> Microsoft Storage Spaces SMP
>> smstsmgr                            manual       stopped
>> ConfigMgr Task Sequence Agent
>> SNMP                                automatic    started        SNMP
>> Service
>> SNMPTRAP                            automatic    started        SNMP
>> Trap
>> Spooler                             automatic    started        Print
>> Spooler
>> sppsvc                              automatic    stopped        Software
>> Protection
>> SSDPSRV                             disabled     stopped        SSDP
>> Discovery
>> SstpSvc                             manual       stopped        Secure
>> Socket Tunneling Protocol Service
>> svsvc                               manual       stopped        Spot
>> Verifier
>> swprv                               manual       stopped
>> Microsoft Software Shadow Copy Provider
>> SysMain                             manual       stopped
>> Superfetch
>> SystemEventsBroker                  automatic    started        System
>> Events Broker
>> TapiSrv                             manual       stopped
>> Telephony
>> TermService                         manual       started        Remote
>> Desktop Services
>> Themes                              automatic    started        Themes
>> THREADORDER                         manual       stopped        Thread
>> Ordering Server
>> TieringEngineService                manual       stopped        Storage
>> Tiers Management
>> TrkWks                              automatic    started
>> Distributed Link Tracking Client
>> TrustedInstaller                    manual       stopped        Windows
>> Modules Installer
>> UALSVC                              automatic    started        User
>> Access Logging Service
>> UI0Detect                           manual       stopped
>> Interactive Services Detection
>> UmRdpService                        manual       started        Remote
>> Desktop Services UserMode Port Redirector
>> upnphost                            disabled     stopped        UPnP
>> Device Host
>> VaultSvc                            manual       stopped
>> Credential Manager
>> vds                                 manual       stopped        Virtual
>> Disk
>> VGAuthService                       automatic    started        VMware
>> Alias Manager and Ticket Service
>> vmicguestinterface                  manual       stopped        Hyper-V
>> Guest Service Interface
>> vmicheartbeat                       manual       stopped        Hyper-V
>> Heartbeat Service
>> vmickvpexchange                     manual       stopped        Hyper-V
>> Data Exchange Service
>> vmicrdv                             manual       stopped        Hyper-V
>> Remote Desktop Virtualization Service
>> vmicshutdown                        manual       stopped        Hyper-V
>> Guest Shutdown Service
>> vmictimesync                        manual       stopped        Hyper-V
>> Time Synchronization Service
>> vmicvss                             manual       stopped        Hyper-V
>> Volume Shadow Copy Requestor
>> VMTools                             automatic    started        VMware
>> Tools
>> vmvss                               manual       stopped        VMware
>> Snapshot Provider
>> VSS                                 manual       stopped        Volume
>> Shadow Copy
>> W32Time                             manual       started        Windows
>> Time
>> w3logsvc                            manual       stopped        W3C
>> Logging Service
>> W3SVC                               automatic    started        World
>> Wide Web Publishing Service
>> WAS                                 manual       started        Windows
>> Process Activation Service
>> Wcmsvc                              automatic    started        Windows
>> Connection Manager
>> WcsPlugInService                    manual       stopped        Windows
>> Color System
>> WdiServiceHost                      manual       stopped
>> Diagnostic Service Host
>> WdiSystemHost                       manual       stopped
>> Diagnostic System Host
>> Wecsvc                              manual       stopped        Windows
>> Event Collector
>> WEPHOSTSVC                          manual       stopped        Windows
>> Encryption Provider Host Service
>> wercplsupport                       manual       stopped        Problem
>> Reports and Solutions Control Panel Support
>> WerSvc                              manual       stopped        Windows
>> Error Reporting Service
>> WinHttpAutoProxySvc                 manual       started        WinHTTP
>> Web Proxy Auto-Discovery Service
>> Winmgmt                             automatic    started        Windows
>> Management Instrumentation
>> WinRM                               automatic    started        Windows
>> Remote Management (WS-Management)
>> wmiApSrv                            manual       stopped        WMI
>> Performance Adapter
>> WPDBusEnum                          manual       stopped        Portable
>> Device Enumerator Service
>> WSService                           manual       stopped        Windows
>> Store Service (WSService)
>> wuauserv                            automatic    started        Windows
>> Update
>> wudfsvc                             manual       stopped        Windows
>> Driver Foundation - User-mode Driver Framework
>>
>>
>> On Mon, Oct 17, 2016 at 4:10 PM, Japheth Cleaver
>> <cleaver at terabithia.org>
>> wrote:
>>
>>> Hmm. Does the data after the corrupted lines appear to match the
>>> remaining data for the server in question? From the sample below it
>>> seems
>>> not (as I believe this is reported in alphabetical order), which might
>>> indicate indicate a broader memory corruption issue going on within
>>> xymond_client, where it's somehow losing track of the end or garbling
>>> the
>>> data in the buffer being used for holding status output. If it's
>>> causing a
>>> false positive, then it's not merely the final output that's the
>>> problem,
>>> but something occurring earlier in processing.
>>>
>>> What OS+distro is the server running on?
>>> Any chance you might be able to run xymond_client in debug mode for a
>>> bit
>>> while this is occurring?
>>>
>>> -jc
>>>
>>>
>>>
>>> On 10/17/2016 7:56 AM, Greg Krpan wrote:
>>>
>>> Hi JC-
>>>
>>> Thanks for the response.
>>>
>>> I am using Xymon 4.3.27 currently.  The raw client data looks fine-
>>> there
>>> are no corrupted lines and no added brackets or special characters that
>>> I
>>> can see.  This only occurs on the status pages.
>>>
>>> The server has been running since May, and this particular problem
>>> started at the end of Sept., after running Windows Update on my
>>> servers,
>>> but as both Windows and Linux clients are showing the behavior, I have
>>> ruled out the updates as the issue.
>>>
>>> I have tried restarting the service with no effect on behavior and
>>> there
>>> is nothing in the log files that show a problem that I can see.  The
>>> level
>>> of false positives due to formatting errors has remained relatively
>>> consistent, and tends to be limited to the PROCS (Win, Linux) and SVCS
>>> (Win
>>> only) tests, but occasionally will see the same error occurring on the
>>> DISK
>>> and CPU tests, although that is significantly less frequent, and is not
>>> across all configured machines.  The PROCS/SVCS tests are showing
>>> random
>>> errors on one machine or another approximately every 5 minutes.
>>>
>>> Thanks
>>> Greg.
>>>
>>> On Fri, Oct 14, 2016 at 6:52 PM, J.C. Cleaver <cleaver at terabithia.org>
>>> wrote:
>>>
>>>>
>>>>
>>>> On Fri, October 14, 2016 3:52 pm, Greg Krpan wrote:
>>>> > Recently, my monitoring has been generating frequent errors that are
>>>> > false,
>>>> > due to improper formatting,  It is happening on both Windows and
>>>> Linux
>>>> > clients.  I've included an example of how the tests are sending data
>>>> back
>>>> > to the xymon server.  I have not made any changes to my client or
>>>> server
>>>> > configurations.  Has anyone else been experiencing this behavior, or
>>>> know
>>>> > of a fix?
>>>> >
>>>> > Greg.
>>>> >
>>>> > Name                                StartupType  Status
>>>> > DisplayName
>>>> > AeLookupSvc                         manual       stopped
>>>> > Application Experience
>>>> > ALG                                 manual       stopped
>>>> > Application Layer Gateway Service
>>>> > AppIDSvc                            manual       stopped
>>>> > Application Identity
>>>> > Appinfo                             manual       stopped
>>>> > Application Information
>>>> > AppMgmt                             manual       stopped
>>>> > Application Management
>>>> > AppReadiness                        manual       stopped        App
>>>> > Readiness
>>>> > AppXSvc                             manual       stopped        AppX
>>>> > Deployment Service (AppXSVC)
>>>> > AudioEndpointBuilder                manual
>>>> > toppe]        Windows Audio Endpoint Builder
>>>> > Audiosrv                            manual       stopped
>>>> Windows
>>>> > Audio
>>>> > BBWin                               automatic    started        Big
>>>> > Brother Xymon Client
>>>> > BFE                                 automatic    started        Base
>>>> > Filtering Engine
>>>> > BITS                                automatic    started
>>>> > Background Intelligent Transfer Serv
>>>> > ce
>>>> > BrokerInfrastructure   ]            automatic    started
>>>> > Background Tasks Infrastructure Service
>>>> > Browser                             disabled     stopped
>>>> Computer
>>>> > Browser
>>>> > CcmExec                             automatic    started        SMS
>>>> Agent
>>>> > Host
>>>> > CertPropSvc                         manual       started
>>>> > Certificate Propagation
>>>> > CmRcService                         disabled     stopped
>>>> > Configuration Manager Remote Control
>>>> > COMSysApp                           manual
>>>> > started        COM+ Sys]
>>>> > m Application
>>>> > CryptSvc]
>>>> >                          ]
>>>> > utomatic    started        Cr]
>>>> > tographic Services
>>>> > DcomLaunch   ]
>>>> >                      automatic    sta]
>>>> > ed        DCOM Serv]
>>>> >  Process Launcher
>>>> > defra]svc                           manual       stopped
>>>> Optimize
>>>> > drives
>>>> > DeviceAssociationService            manual       stopped
>>>> Device
>>>> > Association Service
>>>>
>>>>
>>>> Hi Greg,
>>>>
>>>> Is there anything unusual about the process names on the lines
>>>> immediately
>>>> before the corruption? There's a known issue in that lines starting
>>>> with
>>>> a
>>>> bracket will cause missing data, and this can happen more frequently
>>>> on
>>>> Windows servers just by virtue of some of the data that's coming
>>>> across,
>>>> but that doesn't appear to be causing this specific issue.
>>>>
>>>>
>>>> Can you confirm which version of Xymon server you're using? Do you see
>>>> the
>>>> same corruption in the "raw" Client Data for the affected servers, or
>>>> is
>>>> it only occurring on the status pages?
>>>>
>>>> Also -- anything unusual in the log files? Has this problem been
>>>> constant
>>>> since it started, or is it getting worse? Does restarting the xymon
>>>> service fix it (temporarily)?
>>>>
>>>>
>>>> Regards,
>>>> -jc
>>>>
>>>>
>>>
>>>
>>> --
>>> ---------------------------------------------------------------------------
>>>
>>> In honor of those who lost their lives exploring the final frontier:
>>> Apollo 1; January 27, 1967 Virgil "Gus" Ivan Grissom, Edward Higgins
>>> White II, Roger Bruce Chaffee
>>> Space Shuttle Challenger, Mission STS-51-L; January 28, 1986 Francis R.
>>> Scobee, Michael J. Smith, Judith A. Resnik, Ellison S. Onizuka, Ronald
>>> E.
>>> McNair, Gregory B. Jarvis, Sharon Christa McAuliffe
>>> Space Shuttle Columbia, Mission STS-107; February 1, 2003 Rick D.
>>> Husband, William C. McCool, Michael P. Anderson, Kalpana Chawla, David
>>> M.
>>> Brown, Laurel Blair Salton Clark, Ilan Ramon
>>>
>>>
>>>
>>
>>
>> --
>> ---------------------------------------------------------------------------
>>
>> In honor of those who lost their lives exploring the final frontier:
>> Apollo 1; January 27, 1967 Virgil "Gus" Ivan Grissom, Edward Higgins
>> White
>> II, Roger Bruce Chaffee
>> Space Shuttle Challenger, Mission STS-51-L; January 28, 1986 Francis R.
>> Scobee, Michael J. Smith, Judith A. Resnik, Ellison S. Onizuka, Ronald
>> E.
>> McNair, Gregory B. Jarvis, Sharon Christa McAuliffe
>> Space Shuttle Columbia, Mission STS-107; February 1, 2003 Rick D.
>> Husband,
>> William C. McCool, Michael P. Anderson, Kalpana Chawla, David M. Brown,
>> Laurel Blair Salton Clark, Ilan Ramon
>>
>
>
>
> --
> ---------------------------------------------------------------------------
> In honor of those who lost their lives exploring the final frontier:
> Apollo 1; January 27, 1967 Virgil "Gus" Ivan Grissom, Edward Higgins White
> II, Roger Bruce Chaffee
> Space Shuttle Challenger, Mission STS-51-L; January 28, 1986 Francis R.
> Scobee, Michael J. Smith, Judith A. Resnik, Ellison S. Onizuka, Ronald E.
> McNair, Gregory B. Jarvis, Sharon Christa McAuliffe
> Space Shuttle Columbia, Mission STS-107; February 1, 2003 Rick D. Husband,
> William C. McCool, Michael P. Anderson, Kalpana Chawla, David M. Brown,
> Laurel Blair Salton Clark, Ilan Ramon
>





More information about the Xymon mailing list