[hobbit] Route/Netstat Crashing and/or Ipconfig Not Working = Purple results

Fri Jan 15 17:40:50 CET 2010

Folks
I had enabled most versbose logging in order to see if this helps to
show what is going on one of the servers. This is the output:

009/12/21 14:46:40 [DEBUG]: [svcs]: start AgentSvcs::InitCentralMode
2009/12/21 14:46:40 [DEBUG]: [svcs]: checking file c:\Program
Files\BBWin\tmp\clientlocal.cfg
2009/12/21 14:46:40 [DEBUG]: [svcs]: reading file c:\Program
Files\BBWin\tmp\clientlocal.cfg
2009/12/21 14:49:05 [DEBUG]: [externals]: starting external 'cmd /c
..\ext\termusers\termusers.bat'
2009/12/21 14:51:41 [DEBUG]: [cpu]: cpu review started
2009/12/21 14:51:41 [DEBUG]: [cpu]: cpu review ended
2009/12/21 14:51:41 [DEBUG]: [disk]: drive string C:\
2009/12/21 14:51:41 [DEBUG]: [disk]: drive string D:\
2009/12/21 14:51:41 [DEBUG]: [disk]: drive string E:\
2009/12/21 14:51:41 [DEBUG]: [disk]: drive string V:\
2009/12/21 14:51:41 [DEBUG]: [disk]: drive string X:\
2009/12/21 14:51:41 [DEBUG]: [msgs]: start AgentMsgs::InitCentralMode
2009/12/21 14:51:41 [DEBUG]: [msgs]: checking file c:\Program
Files\BBWin\tmp\clientlocal.cfg
2009/12/21 14:51:41 [DEBUG]: [msgs]: reading file c:\Program
Files\BBWin\tmp\clientlocal.cfg
2009/12/21 14:51:41 [DEBUG]: [msgs]: will use maxdata size : 10240
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore : is an invalid print
stream
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore : permission settings
do not grant
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore : Office Document Image
Writer
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore : warning
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore : Warning
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore : Print
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore : print
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore : pages printed
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore : information
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore : Information
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore : perflib
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore : Perflib
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore : AutoEnrollment
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore : LoadPerf
2009/12/21 14:51:41 [DEBUG]: [msgs]: will use maxdata size : 10240
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore :
BigBrotherHobbitClient
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore :
bigbrotherhobbitclient
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore : information
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore : Information
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore : cccredmgr
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore : userenv
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore : Userenv
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore : warning
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore : Warning
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore : Windows saved user 
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore : Windows cannot unload
your classes registry file
2009/12/21 14:51:41 [DEBUG]: [msgs]: will use maxdata size : 10240
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore : Restricted Sid Count
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore : auditable protected
data
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore : Backup of data
protection master key
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore : success
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore : Success
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore : process
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore : Destination DRA
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore : The description for
Event ID
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore : Handle Closed
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore : Logon attempt
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore : Successful Network
Logon
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore : User Logoff
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore : Service Ticket
Renewed
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore : Service Ticket
Request
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore : Authentication Ticket
Request
2009/12/21 14:51:41 [DEBUG]: [msgs]: will use maxdata size : 10240
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore : Health
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore : health
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore : information
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore : Information
2009/12/21 14:51:41 [DEBUG]: [msgs]: will ignore : failed
2009/12/21 14:52:05 [DEBUG]: [externals]: starting external 'cmd /c
..\ext\termusers\termusers.bat'
2009/12/21 14:55:05 [DEBUG]: [externals]: starting external 'cmd /c
..\ext\termusers\termusers.bat'
2009/12/21 14:58:05 [DEBUG]: [externals]: starting external 'cmd /c
..\ext\termusers\termusers.bat'
2009/12/21 15:01:05 [DEBUG]: [externals]: starting external 'cmd /c
..\ext\termusers\termusers.bat'
2009/12/21 15:04:05 [DEBUG]: [externals]: starting external 'cmd /c
..\ext\termusers\termusers.bat'
2009/12/21 15:07:05 [DEBUG]: [externals]: starting external 'cmd /c
..\ext\termusers\termusers.bat'
2009/12/21 15:10:05 [DEBUG]: [externals]: starting external 'cmd /c
..\ext\termusers\termusers.bat'
2009/12/21 15:13:05 [DEBUG]: [externals]: starting external 'cmd /c
..\ext\termusers\termusers.bat'
2009/12/21 15:16:05 [DEBUG]: [externals]: starting external 'cmd /c
..\ext\termusers\termusers.bat'
2009/12/21 15:19:05 [DEBUG]: [externals]: starting external 'cmd /c
..\ext\termusers\termusers.bat'
2009/12/21 15:22:05 [DEBUG]: [externals]: starting external 'cmd /c
..\ext\termusers\termusers.bat'
2009/12/21 15:25:05 [DEBUG]: [externals]: starting external 'cmd /c
..\ext\termusers\termusers.bat'
2009/12/21 15:28:05 [DEBUG]: [externals]: starting external 'cmd /c
..\ext\termusers\termusers.bat'
2009/12/21 15:31:05 [DEBUG]: [externals]: starting external 'cmd /c
..\ext\termusers\termusers.bat'
2009/12/21 15:34:05 [DEBUG]: [externals]: starting external 'cmd /c
..\ext\termusers\termusers.bat'
2009/12/21 15:37:05 [DEBUG]: [externals]: starting external 'cmd /c
..\ext\termusers\termusers.bat'
2009/12/21 15:40:05 [DEBUG]: [externals]: starting external 'cmd /c
..\ext\termusers\termusers.bat'
2009/12/21 15:43:05 [DEBUG]: [externals]: starting external 'cmd /c
..\ext\termusers\termusers.bat'
2009/12/21 15:46:05 [DEBUG]: [externals]: starting external 'cmd /c
..\ext\termusers\termusers.bat'
2009/12/21 15:49:05 [DEBUG]: [externals]: starting external 'cmd /c
..\ext\termusers\termusers.bat'
2009/12/21 15:52:05 [DEBUG]: [externals]: starting external 'cmd /c
..\ext\termusers\termusers.bat'
2009/12/21 15:55:05 [DEBUG]: [externals]: starting external 'cmd /c
..\ext\termusers\termusers.bat'
2009/12/21 15:58:05 [DEBUG]: [externals]: starting external 'cmd /c
..\ext\termusers\termusers.bat'
2009/12/21 16:01:05 [DEBUG]: [externals]: starting external 'cmd /c
..\ext\termusers\termusers.bat'
2009/12/21 16:04:05 [DEBUG]: [externals]: starting external 'cmd /c
..\ext\termusers\termusers.bat'
2009/12/21 16:07:05 [DEBUG]: [externals]: starting external 'cmd /c
..\ext\termusers\termusers.bat'
2009/12/21 16:10:05 [DEBUG]: [externals]: starting external 'cmd /c
..\ext\termusers\termusers.bat'
2009/12/21 16:13:05 [DEBUG]: [externals]: starting external 'cmd /c
..\ext\termusers\termusers.bat'
2009/12/21 16:16:05 [DEBUG]: [externals]: starting external 'cmd /c
..\ext\termusers\termusers.bat'
2009/12/21 16:19:05 [DEBUG]: [externals]: starting external 'cmd /c
..\ext\termusers\termusers.bat'
2009/12/21 16:22:05 [DEBUG]: [externals]: starting external 'cmd /c
..\ext\termusers\termusers.bat'
2009/12/21 16:25:05 [DEBUG]: [externals]: starting external 'cmd /c
..\ext\termusers\termusers.bat'

Then it just keeps going like that until we correct it. I can try to
disable the external script (that is suppose to report the number of
Terminal users back to hobbit).

I want to double check to make sure I anser the local vrs central mode.

One thing I thought, could something else running on the server cause
issues with hobbit (maybe another tool/script is causing the crash and
holding hobbit up)?

________________________________

From: Jim Sloan [mailto:Jim.Sloan at S1.com] 
Sent: Friday, January 15, 2010 9:50 AM
To: hobbit at hswn.dk
Subject: RE: [hobbit] Route/Netstat Crashing and/or Ipconfig Not Working
= Purple results

Are you running BBWin in local or central mode?

Do you have any external scripts running besides the defaults?

BBWin shouldn't be executing ipconfig or netstat

In the bbwin.cfg change the following line

<setting name="loglevel" value="0">

Change 0 to 1

No need to restart as bbwin will restart on a change of the cfg file.

Jim Sloan

Chief Systems Architect - Corporate IT

S1 Corporation

Tel (direct): +1 404 923 6527

Tel (switchboard): +1 404 923 3500

jim.sloan at s1.com <mailto:jim.sloan at s1.com> 

www.s1.com

Notice: This communication contains information that may be privileged,
confidential and/or exempt from disclosure under applicable law. If you
are not the intended recipient, you are hereby notified that any
disclosure, copying, distribution, or use of the information contained
herein (including any reliance thereon) is strictly prohibited. If you
received this transmission in error, please immediately contact the
sender and delete this communication.

From: Birney Jr Joe-FJB047 [mailto:fjb047 at motorola.com] 
Sent: Friday, January 15, 2010 7:57 AM
To: hobbit at hswn.dk
Subject: RE: [hobbit] Route/Netstat Crashing and/or Ipconfig Not Working
= Purple results

Folks,

all servers are running Win2003 w/SP 2 as the OS.

BBWin is version 0.12

________________________________

From: Odinn [mailto:odinn_asgaard at yahoo.com] 
Sent: Thursday, January 14, 2010 3:11 PM
To: hobbit at hswn.dk
Subject: Re: [hobbit] Route/Netstat Crashing and/or Ipconfig Not Working
= Purple results

what client are you running on the Windows systems? BBWin v.?

--

Jim Sloan

Just remember, today is the day you thought tomorrow was going to be
yesterday. 

________________________________

From: Birney Jr Joe-FJB047 <fjb047 at motorola.com>
To: hobbit at hswn.dk
Sent: Thu, January 14, 2010 2:46:48 PM
Subject: RE: [hobbit] Route/Netstat Crashing and/or Ipconfig Not Working
= Purple results

Thanks for the feedback. We had to install Hobbit on about 40 servers.
We ended up scripting the install to make life easier and ensure all of
the servers got the same version, installed the same day, ect. We also
made sure that the we had set the mode on the citrix box to install
before we ran the script. 

Is there any way that we can get Hobbit to run the commands in more of a
verbose mode? We would love to see the cmd window that it spawns to run
the netstat or ipconfig commands so we can see what its hanging up as...
We tried to change one of the servers so bbwin runs as with a different
account and left that account logged in hoping a CMD window or other
data would be persevered. But no luck :(

Jb

________________________________

From: Odinn [mailto:odinn_asgaard at yahoo.com] 
Sent: Thursday, January 14, 2010 10:57 AM
To: hobbit at hswn.dk
Subject: Re: [hobbit] Route/Netstat Crashing and/or Ipconfig Not Working
= Purple results

This sounds like an issue with the client, not the server.

--

Jim Sloan

Just remember, today is the day you thought tomorrow was going to be
yesterday. 

________________________________

From: Birney Jr Joe-FJB047 <fjb047 at motorola.com>
To: hobbit at hswn.dk
Sent: Thu, January 14, 2010 8:46:16 AM
Subject: [hobbit] Route/Netstat Crashing and/or Ipconfig Not Working =
Purple results

Folks,
we have three dell power edge 2950s that are running a Win2k3 OS as well
as the Citrix Xenapp (version 5.0). What we see is that Hobbit will
report the basic test purple (CPU, Disk, Mem, ect) after some time has
pasted. When we logon to the systems we usually see a crash message
saying that either route.exe or netstat has crashed. Checking the task
manger on these boxes shows we can have a mix of netstat and route.exe
still running (some times its just one, other there are 2 or more
instances, or just one of them running, ect). Looking at the event
viewer we will usually see: 

The agent stats generated this event message : failed to execute
ipconfig /all  

But no other error indications on those boxes. And of course when we run
this ipconfig command it works all the time for us. The simple fix is to
kill any an all netstat.exe, route.exe and cmd windows they tried to run
and then the servers go back to green. Problem is they will only last
for some time until they go purple. And the time it takes is random.
Finally we have 6 other Citrix servers all configured in the same manor
that do NOT exhibit this behavior.

So any ideas on how to find out why netstat/route keep crashing?

Thanks! 
Jb 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xymon.com/pipermail/xymon/attachments/20100115/3d8fb7b8/attachment.html>