<div dir="ltr">As an additional update, I've checked multiple times and the raw data that is passed to the system is correct. It appears to be when it is displayed on the webpage where the problem occurs. I have not updated xymon- it appears as though the version that I am on is still the most recent version (4.3.27) Is anyone aware of any conflicts that may have been introduced after patching? Since this is a production server, I run patches on a monthly basis, with the most recent patching occurring on 9/29, which is coincidentally when the problem started occurring.</div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Oct 17, 2016 at 4:27 PM, Greg Krpan <span dir="ltr"><<a href="mailto:gjkrpan43@gmail.com" target="_blank">gjkrpan43@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I've included an entire "SVCS" status below on a failed status screen. As you can see, it is random as to how and where the output corrupts, <div><br></div><div>On the Windows systems, I run BBWin for the client (version 0.13). I should be able to put the xymond_client process into debug mode to monitor for a while as well.. The problem is less predominant on Linux clients than on Windows, but it is occurring on both.<br><div><br></div><div>The server is running on CentOS 7 with current patches.</div><div><div># uname -a</div><div>Linux ************************* 3.10.0-327.36.1.el7.x86_64 #1 SMP Sun Sep 18 13:04:29 UTC 2016 x86_64 x86_64 x86_64 GNU/Linu</div></div><div><br><div><div># cat /etc/centos-release</div><div>CentOS Linux release 7.2.1511 (Core)</div><div><br></div><div>[# cat /etc/os-release</div><div>NAME="CentOS Linux"</div><div>VERSION="7 (Core)"</div><div>ID="centos"</div><div>ID_LIKE="rhel fedora"</div><div>VERSION_ID="7"</div><div>PRETTY_NAME="CentOS Linux 7 (Core)"</div><div>ANSI_COLOR="0;31"</div><div>CPE_NAME="cpe:/o:centos:<wbr>centos:7"</div><div>HOME_URL="<a href="https://www.centos.org/" target="_blank">https://www.centos.<wbr>org/</a>"</div><div>BUG_REPORT_URL="<a href="https://bugs.centos.org/" target="_blank">https://bugs.<wbr>centos.org/</a>"</div><div><br></div><div>CENTOS_MANTISBT_PROJECT="<wbr>CentOS-7"</div><div>CENTOS_MANTISBT_PROJECT_<wbr>VERSION="7"</div><div>REDHAT_SUPPORT_PRODUCT="<wbr>centos"</div><div>REDHAT_SUPPORT_PRODUCT_<wbr>VERSION="7"</div></div><div><br></div><div><br></div><div><pre style="color:rgb(216,216,191);background-color:rgb(0,0,0)">Name StartupType Status DisplayName
AeLookupSvc manual stopped Application Experience
ALG manual stopped Application Layer Gateway Service
AppHostSvc automatic started Application Host Helper Service
AppIDSvc manual stopped Application Identity
Appinfo manual stopped Application I
forma]ion
AppMgmt manual stopped Application Management
AppReadiness manual stopped App Readiness
AppXSvc manual stopped AppX Deployment Service (AppXSVC)
aspnet_state manual stopped <a href="http://ASP.NET" target="_blank">ASP.NET</a> State Service
AudioEndpointBuilder man
al stopped Wi]dows Audio Endpoint Builder
Audiosrv manual stopped Windows Audio
BBWin automatic started Big Brother Xymon Client
BFE automatic started Base Filtering Engine
BITS automatic started Back
round Intelligent Trans]
r Service
BrokerInfrastru]
ure automatic ]
started Background Tasks]
nfrastructure Service
Browser ]
]isabled stopped Computer Browser
CcmExec automatic started SMS Agent Host
CertPropSvc manual started Certificate Propagation
CmRcService disabled stopped Configuration Manager Remote Control
COMSysApp manual started COM+ System Application
CryptSvc automatic started Cryptographic Services
DcomLaunch automatic started DCOM Server Process Launcher
defragsvc manual stopped Optimize drives
DeviceAssociationService manual stopped Device Association Service
DeviceInstall manual stopped Device Install Service
Dhcp automatic started DHCP Client
DiagTrack automatic started Diagnostics Tracking Service
Dnscache automatic started DNS Client
dot3svc manual stopped Wired AutoConfig
DPS automatic started Diagnostic Policy Service
DsmSvc manual started Device Setup Manager
Eaphost manual stopped Extensible Authentication Protocol
EFS manual stopped Encrypting File System (EFS)
EventLog automatic started Windows Event Log
EventSystem automatic started COM+ Event System
fdPHost manual stopped Function Discovery Provider Host
FDResPub manual stopped Function Discovery Resource Publication
FontCache automatic started Windows Font Cache Service
gpsvc automatic started Group Policy Client
hidserv manual stopped Human Interface Device Service
hkmsvc manual stopped Health Key and Certificate Management
IEEtwCollectorService manual stopped Internet Explorer ETW Collector Service
IISADMIN automatic started IIS Admin Service
IKEEXT automatic started IKE and AuthIP IPsec Keying Modules
iphlpsvc automatic started IP Helper
KeyIso manual started CNG Key Isolation
KPSSVC manual stopped KDC Proxy Server service (KPS)
KtmRm manual stopped KtmRm for Distributed Transaction Coordinator
LanmanServer automatic started Server
LanmanWorkstation automatic started Workstation
lltdsvc manual stopped Link-Layer Topology Discovery Mapper
lmhosts automatic started TCP/IP NetBIOS Helper
lpasvc manual stopped Microsoft Policy Platform Local Authority
lppsvc manual stopped Microsoft Policy Platform Processor
LSM automatic started Local Session Manager
McAfeeFramework automatic started McAfee Framework Service
McShield automatic started McAfee McShield
McTaskManager automatic started McAfee Task Manager
MMCSS manual stopped Multimedia Class Scheduler
MpsSvc
automatic started ] Windows Firewall
MSDTC automatic started Distributed Transaction Coordinator
MSiSCSI manual stopped Microsoft iSCSI Initiator Service
msiserver manual stopped Windows Installer
napagent manual stopped Network Access Protection Agent
NcaSvc manual stopped Network Connectivity Assistant
Netlogon automatic started Netlogon
Netman manual stopped Network Connections
netprofm manual started Network List Service
NetTcpPortSharing disabled stopped Net.Tcp Port Sharing Service
NlaSvc automatic started Network Location Awareness
nsi automatic started Network Store Interface Service
PerfHost manual stopped Performance Counter DLL Host
pla manual stopped Performance Logs & Alerts
PlugPlay manual started Plug and Play
PolicyAgent manual started IPsec Policy Agent
Power automatic started Power
PrintNotify manual stopped Printer Extensions and Notifications
ProfSvc automatic started User Profile Service
QBCFMonitorService automatic started QuickBooks Database Manager Service
QBFCService manual stopped Intuit QuickBooks FCS
QuickBooksDB17 automatic started QuickBooksDB17
RasAuto manual stopped Remote Access Auto Connection Manager
RasMan manual stopped Remote Access Connection Manager
RemoteAccess disabled stopped Routing and Remote Access
RemoteRegistry automatic stopped Remote Registry
RpcEptMapper automatic started RPC Endpoint Mapper
RpcLocator manual stopped Remote Procedure Call (RPC) Locator
RpcSs automatic started Remote Procedure Call (RPC)
RSoPProv manual stopped Resultant Set of Policy Provider
sacsvr manual stopped Special Administration Console Helper
SamSs automatic started Security Accounts Manager
SCardSvr disabled stopped Smart Card
ScDeviceEnum manual stopped Smart Card Device Enumeration Service
Schedule automatic started Task Scheduler
SCPolicySvc manual stopped Smart Card Removal Policy
seclogon manual stopped Secondary Logon
SENS automatic started System Event Notification Service
SessionEnv manual started Remote Desktop Configuration
SharedAccess disabled stopped Internet Connection Sharing (ICS)
ShellHWDetection automatic stopped Shell Hardware Detection
smphost manual stopped Microsoft Storage Spaces SMP
smstsmgr manual stopped ConfigMgr Task Sequence Agent
SNMP automatic started SNMP Service
SNMPTRAP automatic started SNMP Trap
Spooler automatic started Print Spooler
sppsvc automatic stopped Software Protection
SSDPSRV disabled stopped SSDP Discovery
SstpSvc manual stopped Secure Socket Tunneling Protocol Service
svsvc manual stopped Spot Verifier
swprv manual stopped Microsoft Software Shadow Copy Provider
SysMain manual stopped Superfetch
SystemEventsBroker automatic started System Events Broker
TapiSrv manual stopped Telephony
TermService manual started Remote Desktop Services
Themes automatic started Themes
THREADORDER manual stopped Thread Ordering Server
TieringEngineService manual stopped Storage Tiers Management
TrkWks automatic started Distributed Link Tracking Client
TrustedInstaller manual stopped Windows Modules Installer
UALSVC automatic started User Access Logging Service
UI0Detect manual stopped Interactive Services Detection
UmRdpService manual started Remote Desktop Services UserMode Port Redirector
upnphost disabled stopped UPnP Device Host
VaultSvc manual stopped Credential Manager
vds manual stopped Virtual Disk
VGAuthService automatic started VMware Alias Manager and Ticket Service
vmicguestinterface manual stopped Hyper-V Guest Service Interface
vmicheartbeat manual stopped Hyper-V Heartbeat Service
vmickvpexchange manual stopped Hyper-V Data Exchange Service
vmicrdv manual stopped Hyper-V Remote Desktop Virtualization Service
vmicshutdown manual stopped Hyper-V Guest Shutdown Service
vmictimesync manual stopped Hyper-V Time Synchronization Service
vmicvss manual stopped Hyper-V Volume Shadow Copy Requestor
VMTools automatic started VMware Tools
vmvss manual stopped VMware Snapshot Provider
VSS manual stopped Volume Shadow Copy
W32Time manual started Windows Time
w3logsvc manual stopped W3C Logging Service
W3SVC automatic started World Wide Web Publishing Service
WAS manual started Windows Process Activation Service
Wcmsvc automatic started Windows Connection Manager
WcsPlugInService manual stopped Windows Color System
WdiServiceHost manual stopped Diagnostic Service Host
WdiSystemHost manual stopped Diagnostic System Host
Wecsvc manual stopped Windows Event Collector
WEPHOSTSVC manual stopped Windows Encryption Provider Host Service
wercplsupport manual stopped Problem Reports and Solutions Control Panel Support
WerSvc manual stopped Windows Error Reporting Service
WinHttpAutoProxySvc manual started WinHTTP Web Proxy Auto-Discovery Service
Winmgmt automatic started Windows Management Instrumentation
WinRM automatic started Windows Remote Management (WS-Management)
wmiApSrv manual stopped WMI Performance Adapter
WPDBusEnum manual stopped Portable Device Enumerator Service
WSService manual stopped Windows Store Service (WSService)
wuauserv automatic started Windows Update
wudfsvc manual stopped Windows Driver Foundation - User-mode Driver Framework</pre></div></div></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Oct 17, 2016 at 4:10 PM, Japheth Cleaver <span dir="ltr"><<a href="mailto:cleaver@terabithia.org" target="_blank">cleaver@terabithia.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div class="m_5811190531407854370m_534390640183201937moz-cite-prefix">Hmm. Does the data after the corrupted
lines appear to match the remaining data for the server in
question? From the sample below it seems not (as I believe this is
reported in alphabetical order), which might indicate indicate a
broader memory corruption issue going on within xymond_client,
where it's somehow losing track of the end or garbling the data in
the buffer being used for holding status output. If it's causing a
false positive, then it's not merely the final output that's the
problem, but something occurring earlier in processing.<br>
<br>
What OS+distro is the server running on?<br>
Any chance you might be able to run xymond_client in debug mode
for a bit while this is occurring?<span class="m_5811190531407854370HOEnZb"><font color="#888888"><br>
<br>
-jc</font></span><div><div class="m_5811190531407854370h5"><br>
<br>
<br>
On 10/17/2016 7:56 AM, Greg Krpan wrote:<br>
</div></div></div><div><div class="m_5811190531407854370h5">
<blockquote type="cite">
<div dir="ltr">Hi JC-
<div><br>
</div>
<div>Thanks for the response.</div>
<div><br>
</div>
<div>I am using Xymon 4.3.27 currently. The raw client data
looks fine- there are no corrupted lines and no added brackets
or special characters that I can see. This only occurs on the
status pages.</div>
<div><br>
</div>
<div>The server has been running since May, and this particular
problem started at the end of Sept., after running Windows
Update on my servers, but as both Windows and Linux clients
are showing the behavior, I have ruled out the updates as the
issue.</div>
<div><br>
</div>
<div>I have tried restarting the service with no effect on
behavior and there is nothing in the log files that show a
problem that I can see. The level of false positives due to
formatting errors has remained relatively consistent, and
tends to be limited to the PROCS (Win, Linux) and SVCS (Win
only) tests, but occasionally will see the same error
occurring on the DISK and CPU tests, although that is
significantly less frequent, and is not across all configured
machines. The PROCS/SVCS tests are showing random errors on
one machine or another approximately every 5 minutes.</div>
<div><br>
</div>
<div>Thanks<br>
Greg.</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, Oct 14, 2016 at 6:52 PM, J.C.
Cleaver <span dir="ltr"><<a href="mailto:cleaver@terabithia.org" target="_blank">cleaver@terabithia.org</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="m_5811190531407854370m_534390640183201937HOEnZb">
<div class="m_5811190531407854370m_534390640183201937h5"><br>
<br>
On Fri, October 14, 2016 3:52 pm, Greg Krpan wrote:<br>
> Recently, my monitoring has been generating
frequent errors that are<br>
> false,<br>
> due to improper formatting, It is happening on
both Windows and Linux<br>
> clients. I've included an example of how the
tests are sending data back<br>
> to the xymon server. I have not made any changes
to my client or server<br>
> configurations. Has anyone else been
experiencing this behavior, or know<br>
> of a fix?<br>
><br>
> Greg.<br>
><br>
> Name StartupType
Status<br>
> DisplayName<br>
> AeLookupSvc manual
stopped<br>
> Application Experience<br>
> ALG manual
stopped<br>
> Application Layer Gateway Service<br>
> AppIDSvc manual
stopped<br>
> Application Identity<br>
> Appinfo manual
stopped<br>
> Application Information<br>
> AppMgmt manual
stopped<br>
> Application Management<br>
> AppReadiness manual
stopped App<br>
> Readiness<br>
> AppXSvc manual
stopped AppX<br>
> Deployment Service (AppXSVC)<br>
> AudioEndpointBuilder manual<br>
> toppe] Windows Audio Endpoint Builder<br>
> Audiosrv manual
stopped Windows<br>
> Audio<br>
> BBWin automatic
started Big<br>
> Brother Xymon Client<br>
> BFE automatic
started Base<br>
> Filtering Engine<br>
> BITS automatic
started<br>
> Background Intelligent Transfer Serv<br>
> ce<br>
> BrokerInfrastructure ] automatic
started<br>
> Background Tasks Infrastructure Service<br>
> Browser disabled
stopped Computer<br>
> Browser<br>
> CcmExec automatic
started SMS Agent<br>
> Host<br>
> CertPropSvc manual
started<br>
> Certificate Propagation<br>
> CmRcService disabled
stopped<br>
> Configuration Manager Remote Control<br>
> COMSysApp manual<br>
> started COM+ Sys]<br>
> m Application<br>
> CryptSvc]<br>
> ]<br>
> utomatic started Cr]<br>
> tographic Services<br>
> DcomLaunch ]<br>
> automatic sta]<br>
> ed DCOM Serv]<br>
> Process Launcher<br>
> defra]svc manual
stopped Optimize<br>
> drives<br>
> DeviceAssociationService manual
stopped Device<br>
> Association Service<br>
<br>
<br>
</div>
</div>
Hi Greg,<br>
<br>
Is there anything unusual about the process names on the
lines immediately<br>
before the corruption? There's a known issue in that lines
starting with a<br>
bracket will cause missing data, and this can happen more
frequently on<br>
Windows servers just by virtue of some of the data that's
coming across,<br>
but that doesn't appear to be causing this specific issue.<br>
<br>
<br>
Can you confirm which version of Xymon server you're
using? Do you see the<br>
same corruption in the "raw" Client Data for the affected
servers, or is<br>
it only occurring on the status pages?<br>
<br>
Also -- anything unusual in the log files? Has this
problem been constant<br>
since it started, or is it getting worse? Does restarting
the xymon<br>
service fix it (temporarily)?<br>
<br>
<br>
Regards,<br>
-jc<br>
<br>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div class="m_5811190531407854370m_534390640183201937gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">
<div>------------------------------<wbr>------------------------------<wbr>---------------
<br>
In honor of those who lost their lives exploring the
final frontier: <br>
Apollo 1; January 27, 1967 Virgil "Gus" Ivan Grissom,
Edward Higgins White II, Roger Bruce Chaffee <br>
</div>
<div style="text-align:left">Space Shuttle Challenger,
Mission STS-51-L; January 28, 1986 Francis R. Scobee,
Michael J. Smith, Judith A. Resnik, Ellison S. Onizuka,
Ronald E. McNair, Gregory B. Jarvis, Sharon Christa
McAuliffe <br>
</div>
<div>Space Shuttle Columbia, Mission STS-107; February 1,
2003 Rick D. Husband, William C. McCool, Michael P.
Anderson, Kalpana Chawla, David M. Brown, Laurel Blair
Salton Clark, Ilan Ramon<br>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<p><br>
</p>
</div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="m_5811190531407854370gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>------------------------------<wbr>------------------------------<wbr>--------------- <br>In honor of those who lost their lives exploring the final frontier: <br>Apollo 1; January 27, 1967 Virgil "Gus" Ivan Grissom, Edward Higgins White II, Roger Bruce Chaffee <br></div><div style="text-align:left">Space Shuttle Challenger, Mission STS-51-L; January 28, 1986 Francis R. Scobee, Michael J. Smith, Judith A. Resnik, Ellison S. Onizuka, Ronald E. McNair, Gregory B. Jarvis, Sharon Christa McAuliffe <br></div><div>Space Shuttle Columbia, Mission STS-107; February 1, 2003 Rick D. Husband, William C. McCool, Michael P. Anderson, Kalpana Chawla, David M. Brown, Laurel Blair Salton Clark, Ilan Ramon<br></div></div></div>
</div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>--------------------------------------------------------------------------- <br>In honor of those who lost their lives exploring the final frontier: <br>Apollo 1; January 27, 1967 Virgil "Gus" Ivan Grissom, Edward Higgins White II, Roger Bruce Chaffee <br></div><div style="text-align:left">Space Shuttle Challenger, Mission STS-51-L; January 28, 1986 Francis R. Scobee, Michael J. Smith, Judith A. Resnik, Ellison S. Onizuka, Ronald E. McNair, Gregory B. Jarvis, Sharon Christa McAuliffe <br></div><div>Space Shuttle Columbia, Mission STS-107; February 1, 2003 Rick D. Husband, William C. McCool, Michael P. Anderson, Kalpana Chawla, David M. Brown, Laurel Blair Salton Clark, Ilan Ramon<br></div></div></div>
</div>