BBNT An NT/W2K **CLIENT** for Big Brother System & Network Monitor Version 1.08d Questions/suggestions/bugs ? Contact: Quest Software, Inc E-mail: support@bb4.com If you require support, please join one of the mailing lists. For more details, consult http://bb4.com/support.html This client runs on NT 4.0 (SP4 or higher is recommended), Windows 2000 and Windows 2003. This client REQUIRES that you already have installed the "Big Brother System & Network Monitor" server package either on Unix or NT/W2K/W2003 PLEASE SEE NOTE AT BOTTOM FOR MEMORY LEAK PROBLEMS. ALSO SEE NOTE IF SPURRIOUS EVENT LOGS ARE GENERATED BY NT PERFORMANCE INFORMATION RETRIEVAL *************** * * READ THIS IF YOU ARE UPGRADING FROM VERSION 1.06b AND LESS * * YOU MUST REINSTALL AS SERVICE AND REGISTRY LOCATIONS HAVE CHANGED * FIRST REMOVE BBNT USING THE OLD BBNT EXECUTABLE * THEN INSTALL WITH THE NEW EXECUTABLE * SO IF THIS APPLIES TO YOU, DO IT NOW !!! * * CD * BBNT -REMOVE * * SORRY FOR THE INCONVENIENCE * *************** History: 1.08d) Jan 20th 2004 Fixed a bug where in certain cases an upgrade from a previous version would fail. 1.08c) Oct 15th 2003 In CPU report, memory statistics are displayed in either MB or GB A notification request is sent to the BBPAGER based on the color defined in PageLevels (same as Unix client and Unix/Windows server) Procs check now sends out a "No processes to check" status report if no processes are defined to be checked License Modification 1.08b) Oct 27th 2001 bbntcfg.exe can now set ForceRegClose key Performance counters index hardcoded if can't they be determined. Fixed "Version mismatch" error in bbntcfg.exe Messages to BBDISPLAYs/BBPAGERs are now threaded (won't bog down BBNT if they can't be reached) Services names with ':' in their names should be handled properly. Improved uninstall 1.08a) skipped 1.08) Sep 26th 2001 Added launching of external programs Added check for services if in running/stopped state Enchanced bbntcfg: Better error messages Stop/Start of the BBNT service for the current host http://bb4.com can be viewed by clicking on the BB icon BBDISPLAY/BBPAGER can be defined as an IP address or a hostname Added "Externals" and "Services" keys 1.07g) Reduced # of tries to connect to registry before using default key Alternate hostname can be specified in a plug-in status log (external path feature) Fixed bbntcfg.exe "Version Mismatch" bug on W2K 1.07f) Fixed Daylight Savings Time bug caused by broken C run-time library in MSVC++ 6.0 (Only on Intel platforms, no fix for Alpha from MS) 1.07e) Added check to disable cpu/procs check if invalid data was returned by NT kernel. More efficient event log check Added more debug messages 1.07d) Fixed incorrect process load % when installed on a multi-processor system 1.07c) Fixed startup problems on reboot A single process cannot take more than 100% of CPU ;) Sum of %percentages should approximately be 100% External scripts can now set an expiry delay on a status to override the default value on the BBDISPLAY. 1.07b) Packaged with InstallShield Update of the documentation Allows process list in config editor fro remote configuration Fixed some idiosyncarices with config editor Minor changes on how performance data is retrieved Display "Machine recently booted" message in CPU status for one hour after a reboot (just like on a Unix host) Added Memory statistics and Most active Processes to CPU status Fixed Physical memory display when memory > 2G 1.07a) Had mistakenly disabled plugins, re-enabled them 1.07) WSAStartup/WSACleanup/gethostbyname within the same loop it seems it would sometimes send NT into a memory leak frenzy... WSAStartup & WSACleanup only called once FQDN flag was not checked against the uppercase value such that 'y' was taken to be non-fqdn, only the 'Y' would make it into FQDN... Process names are case insensitive Change BBNT NT Service name Moved location of BBNT Registry variables Added "ForceRegClose" registry key 1.06b) That was the beta of some changes in 1.07 except for the change in the BBNT service name & the registry location 1.06a) Remove notification backward compatibility with old style notifications (BB pre-1.08b server) Fixed bug in notification for a plugin 1.06) Multiple BBDISPLAY & BBPAGERS allowed (seperated by ';') 1.05a) Some code cleanup ActivateLog can be set on and off 1.05) Added Timer for plugins Added Alias to replace NT's IP host name Return Procs & Disk like the U*x version You can check that a process does not run A plugin status file is not removed if the name is prefixed with '-' 1.04f) Fixed External plugin bug where only the first file would be sent across Also when using bbntcfg, grey out the disk drop-down box when configuring across the network If you are installing BBNT using the InstallShield Installer, you can skip this section 1) To install BBNT manually: As an administrative user Unzip the bbnt.zip file into any directory (on a LOCAL drive !!!) Start a console window CD to that directory and then execute BBNT with the following arguments: bbnt [-y] -install BBDISPLAY FQDN IPPORT BBDISPLAY: IP address of the BB display server FQDN: Return the Fully qualified domain name of the station (Y or N) IPPORT: Port used for communication between BB client and BB server These variables are in the registry at HKEY_LOCAL_MACHINE\SOFTWARE\BigBrother\BBNT BBNT has been installed as a service: check in Control Panel -> Services See "Register variables" for more info on these and other variables. e.g.: bbnt -install 201.201.201.201 N 1984 or bbnt -install 201.201.201.201 (This will set 'Y' for FQDN and 1984 for IPPORT as defaults) or bbnt -y -install 201.201.201.201 (Same as previous but automatically agrees to license agreement which appears at the top of this README file) Then you must start BBNT from the services applet in Control Panel Start->Settings->Control Panel->Services->BBNT->start IMPORTANT: YOU MUST SPECIFY AN ADMINISTRATIVE ACCOUNT TO RUN THE SERVICE IF YOU ARE GOING TO MONITOR NETWORK DRIVES Control Panel -> Services Select Big Brother Click on Startup on the right-hand side In the "Log On As:" window specify "This Account" and put in an administrative account with password Make sure that the bbnt.exe file is only readable/executable/writable by an administrative account 2) To upgrade BBNT manually: IT DOES NOT CHECK IF AN INSTANCE IS ALREADY RUNNING. IF YOU DON'T STOP IT, THE SERVICE WILL USE THE NEW EXECUTABLE ONLY AT NEXT SERVICE RESTART (MANUALLY OR REBOOT) As an administrative user Stop the current instance of BBNT: Control Panel -> Services Backup the previous version ! Unzip the bbnt.zip file into any directory (you can use the previous installation directory). Start a console window CD to that directory and the execute BBNT with the following arguments: bbnt -upgrade It will automatically remove registry variables that are not required for the new version. It will also create new variables that are required. Restart BBNT: Control Panel -> Services 3) To remove BBNT just type at the prompt bbnt -remove This removes all keys in the registry and removes the BBNT service 4) To run BBNT: in Control Panel -> Services, select BB and start or in a batch file: net start bigbrotherclient or using bbntcfg.exe, click on start/stop service button Registry Variables: (A GUI utility is provided to edit these fields: bbntcfg) Activatelog: (Y or N) - Sends some debug output to a file named BB.LOG The log is saved where the TEMP env variable points (if TEMP isn't there then it uses TMP and if it's not there then it puts it in C:\TEMP). AliasName: Put in a name to override the IP host name found in TCP/IP properties. BBDISPLAY: IP Address of BB display server(s) BBPAGE: IP Address of BB pager server(s) CPUalwaysGreen: (Y or N) - CPU test always returns GREEN Defaults: Default thresholds for CPU & DISK By default CPU is 80:95 & DISK is 90:95 format is "service:yellow:red" Values are in the 1-100 range, anything else will be reset to system default. i.e. CPU:75:85 CPU:75:95 DISK:80:90 Also, drives with Yellow/Red status will be marked with * (yellow) or ** (red) identifiers DISKalwaysGreen: (Y or N) - DISK test always returns GREEN DiskList: List of drives with different yellow/red thresholds than defaults (90/95) i.e. D:98:99 G:93:98 L Z:101:101 drive:yellow:red Only FIXED drives are checked automatically Other drives are checked for values only if they are defined in this list !!! (You don't want 50 workstations turning to red because a server went red). In this case drive G,L,Z are checked with L using defaults (Z will never go yellow or red) Syntax checking is very loose so be carefull EventsalwaysGreen: (Y or N) - Event log test always returns GREEN N.B. It still returns the events log entries that match the "Msg Levels" field entries. The equivalent would be achieved by removing all entries in the "Msg Levels". Or by specifying values of 0 minutes in the entries in the "Msg Levels" field. ExternalPath: Directory where external programs can save BB status messages to be sent by BBNT. This key is set with the value entered in the "Saved Log Location" field using the bbntcfg utility. This acts as a plugin facility where external program create their own status messages. Files that have no extensions (the file name should be the service name) are sent to BB. So first create your file with an extension and when it's ready to be sent over then rename it to the service name only. After processing, the status file is removed. Remember to put the following pattern [] in the status file. Basic style paging (pre BB 1.08b) will receive the message starting at [. Note that if the file is prefixed by '-' (-backup) then the file will NOT removed after processing. This is useful if the status is not generated very 5 minutes, it prevents the BB display to turn the corresponding dot to purple because an update hasn't been sent in the last 30 minutes. e.g. this is the content of the status file: red Thu 08 21:10:24 1998 [xxx.domain.com] blah blah blah blah I use the [] to insert a system identifier such that DHCP stations are easily recognized. I strongly suggest you do the same thing when creating your own status messages regardless of paging method. *** You must *** update the SVCERRLIST token with the service that contains all the services:ids combination in the bbwarnsetup.cfg file on the BBPAGER host(s). Failure to do so will result in a ERR(999) code in the notification message. You can set the expiry delay of the status log to overide the default value on the BBDISPLAY. The expiry delay specifies how long before a status log on BBDISPLAY turns to purple if no subsequent status is received for the host.service combination: e.g. Don't turn purple for 26 hours green+26h Thu 08 21:10:24 1998 [xxx.domain.com] blah blah blah blah The 26h specifies 26 hours. Valid formats are 99999[mhd] where m - minutes h - hours d - days This is useful for scripts that you want to execute only once a day: backup checks, daily system checks, etc... You can also use an alternate hostname instead of the default which is the local hostname. This is useful if you are running test for multiple hosts from the same BBNT server. e.g. Send to some.host.anywhere some.host.anywhere:red Thu 08 21:10:24 1998 [xxx.domain.com] blah blah blah blah Externals: This is a ';' delimited list of scripts to be started by BBNT. By default, it will run each script every 300 seconds. You can change the interval by adding an /INT=XXXX value after the script name. i.e. Run TEST.BAT every 300secs while TEST1.BAT runs every 600secs C:\TEMP\TEST1.BAT /INT=600;C:\TEMP\TEST.BAT Note that these scripts must write their status logs in the directory note by the "Saved logs location" in bbntcfg.exe (ExternalPath key in the registry). Also note that you can specify arguments to your script name, i.e. C:\TEMP\TEST1.BAT -Q -R f /INT=600;C:\TEMP\TEST.BAT FQDN: (Y or N) - Return host name as Fully Qualified Domain Name or not FQDN: my.host.com Not FQDN: my ForceRegClose: (Y or N) - This is an internal toggle switch to force the close of the Performance registry key. It defaults to Y. If you experience Dr Watson dumps, set it to N. It may help. IgnoreMsgs: Ignore the event log messages that contain this text. You can also specify an event source. Message will be checked with the text squeezed and ignoring case. If you make a type you are out of luck ! Multiple messages can be defined: seperate each msg with a ';' i.e. Remote Access; Access to performance data If the message text/event source contains either strings then it will not be return in the status message. Note, the comparison is case insensitive and spaces are ignored. A maximum of 2048 characters for all messages is allowed Key Code: If you wish to send the data across to the BBDISPLAY/BBPAGER host(s) then enter the shared secret key in this field. IPport: Ip port for communication between client and server MsgLevels: Events logs checking Format of event log entries associated with a type level Message source: SEC - Security SYS - System APP - Applications Message Level: ERR - Error WARN - Warning INFO - Informational SUCCESS_AUDIT - Audit success FAIL_AUDIT - Audit fail Additional specifiers are Y/N Y (red) / N (Yellow) Elapsed time: How many minutes before ignoring msg (default 30 mins) i.e. SYS:ERR:Y:30 SYS:WARN:N:15 APP:ERR:Y:30 APP:WARN:N:15 If a Source:Level pair is not specified then it is ignored Memlevels: Physical & Total Memory thresholds. Total Memory is physical + swap memory. The format of the field is PHYS:AA:BB COMMIT:CC:DD PHYS defines at which % of physical memory usage will trigger a warning or panic condition. If you don't want to use this threshold, set the warning and panic values to a number greater than 100. AA is the warning level while BB is the panic level. TOTAL defines at which % of physical+swap memory usage will trigger a warning or panic condition. If you don't want to use this threshold, set the warning and panic values to a number greater than 100. CC is the warning level while DD is the panic level. Pagelevels: Which status color will generate a notification request to the BBPAGER. Default is "red purple" PluginTimer: Waiting period between plugins status files loading Default is 60 seconds This means that BBNT waits XXX seconds before checking if any status files were written to the plugin directory (the ExternalPath variable) by plugin programs Procs: List of process names to check if they are (not) running i.e snmp smtp space/tab is the delimiter between process names. Process names are typed without their extention. N.B. process with .com may require full name. Set ActivateLog to Y and check process list to see how the process name should be entered (or use task manager or pview). This key cannot exceed 1024 characters in length. This does not check for correct behavior (the proc might be running but it might be totally screwed up) Extra qualifiers are possible for each process: smtpproc:Y:3 If you put '0' as the number of instances then it will check that the process does NOT run Y will set to red / N will set to yellow SendPageAlerts: Send notification request when color in status is defined in PageLevels (Y/N, Y is default). Services: This ';' delimited list contains the NT services to monitor. You pick the services using the bbntcfg utility. It will use the display name as a reference. Three settings define how the BBNT program processes the services. These settings are appened to the service name. The settings are: 1) Color returned of service does not match setting #2 2) Service state expected - R (running) or S (stopped) 3) Reset service if wrong - Y (yes) or N (no) i.e. DHCP Client:Y:S:N;Big Brother SNM Server 2.2h:R:R:Y DHCP Client: turn yellow if service is not stopped and don't stop if running Big Brother SNM Server 2.2h: turn red if service is not running and restart it if it is not running Timer: Waiting period between checks in seconds Default is 300 seconds NOTICE: ------- The hostname is given in the status message to aid admins with DHCP stations BB does not have to be restarted when changing the registry variables with regedit or the config editor. Variables are reloaded everytime. CPU test: The % is for the last 5 minutes PhysicalMem - in MB - the value in parentheses is % used DISK test: Local drives are alerted on. Remote drives are only alerted if they are specified in DiskList registry variable It only returns CPU, DISK, PROCS and MSGS information (there a lot more on the U*x side) Empty Procs registry variable will not return anything If processes were previously checked and you remove all processes from the process list to check, then you will get a purple condition after 30 minutes, so you better remove the host.procs file in the $BBHOME/www/logs directory on the BBDISPLAY server(s) If your NT workstation is DHCP enabled you will have to set the the host's entry in the bb-hosts file of the BBDISPLAY/BBPAGER/BBNET server(s) to: 0.0.0.0 some.name.com # In the CPU test it always returns 1 user (i think, never really tested this)... When using bbntcfg, the HOST field is the NT name not the IP name (You should always set both to the same name). Alias may be used to force another name than the one found in TCP/IP properties DIGITAL ALPHA USERS: -------------------- Note that there are no bbntcfg.exe for alpha. Use bbntcfg on an x86 platform and configure across the network. Or install FX!32 from Digital (it's an X86 emulator and it's free) and run it directly on the Alpha box. IF YOU DON'T SEE ANY STATUS ON YOUR BBDISPLAY: ---------------------------------------------- Make sure the client is running. Verify that the hostname given in the bb-hosts file of your BBDISPLAY matches the real hostname or the alias given in the "alias" field while configuring with bbntcfg.exe. You can also find out the hostname by looking up in the bbvar/logs directly for the appropriate log. MEMORY LEAKS: ------------- I have (and others) noticed memory leaks depending on the server which is running BB. These leaks are caused by performance extensions in services. You will have to disable all extensions by renaming them. Then reenable them one by one until you find the culprit(s) which you will disable. It also has been reported that BBNT does not start because of 3rd party performace DLLs. Using Regedit go to HKEY_LOCAL_MACHINE/SYSTEM/CurrenttControlSet/Services. Search for the string "Performance". Rename to xPerformance. Set the Timer in BB to 1 to accelerate the refresh. At this point the memory leak should have dissapeared. Reenable one performance extension at a time. Wait a while (15 minutes) and check using Task manager if the memory usage of BBNT has increased. If so, disable the performance extension. Don't forget to reset the refresh to 300 seconds... I strongly suggest you read article Q178887 at MicroSoft support (MSDN) http://support.microsoft.com/support/kb/articles/q178/8/87.asp for more info on the memory leaks problem. In our experience We've had problems with tapiperf.dll, rasman.dll, perfctrs.dll on both X86 and AXP platforms. They are a real pain in the ... PERFLIB ERRORS: --------------- Sometimes some error message similar to this: "The Collect Procedure for the service in DLL generated an exception or returned an invalid status. Performance data returned by counter DLL will not be returned in Perf Data Block. Exception or status code returned is DWORD 0." will be found in the event log. Related reading: http://support.microsoft.com/support/kb/articles/Q201/9/84.asp Thanks to Henry Cowie for the previous link. This is due to an error in the performance extension in the specified DLL. To suppress any of those messages to prevent clogging up your event log, you must change a value in the registry. Edit the key named HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Perflib Change the value of EventLogLevel to 0. If it does not exists, create it vith a value of 0. Related readings: http://support.microsoft.com/support/kb/articles/Q226/4/94.ASP http://support.microsoft.com/support/kb/articles/Q178/8/87.ASP Thanks to Florian Kolbe for this trick. On certain occasions, on Windows 2000 servers, the NT kernel does not seen to return proper performance data. If you don't have cpu/procs status logs for a certain host then it's probably for this reason. To double check, set ActivateLog on and if the "numprocs" or "oldnumprocs" is very high then your server cannot reliably return performance data. Please contact support@bb4.com if you suffer from this problem. Dr Watson Errors: ----------------- If you get DR WATSON errors, set the ForceRegClose registry entry in HKEY_LOCAL_MACHINE\SOFTWARE\BigBrother\bbnt to N. It seems that the Performance registry key requires closing, even if it is specified in Microsoft documentation that you do not have to, because menory leaks may occur. But, this behavior, sometimes, makes BBNT crash: often when Oracle runs on the same server or the server is in a cluster. If BBNT crashes, try setting the ForceRegClose key to N, it may help. Conflicts with Oracle: ---------------------- BBNT and Oracle conflict over performance statistics when Oracle's Performance monitor is installed. To remove this utility, follow these instructions: a. Go to Start|Programs|Oracle For Windows NT|Oracle Installer b. Software Asset Manager screen comes up c. Use the list of installed software (on the right side of the dialog) and select the Oracle Performance Utility d. Click Remove button e. Confirmation dialog comes up, Click Yes f. Exit the Oracle Installer *** Note that the renaming Performance to xPerformance trick also works *** Conflicts with Lotus Notes: --------------------------- BBNT is also prone to conflicts with Lotus Notes. It has been reported that Compaq Insight Manager and IBM Netfinity Manager also suffer the same fate. To remedy the situation, you must first determine if performance statistics is installed in Domino. Start the NT Performance Monitor Tool and check for an Object called "Lotus Notes". To remove this object go to www.support.lotus.com and search for "notes stat" there you should find a document called "How to Remove the Notestat Key from the Windows NT Registry" - document # 174802. Thanks to Paul Wilson for the tip. *** Note that the renaming Performance to xPerformance trick also works *** Terminal Server notes: ---------------------- This applies to: . Windows NT Server 4.0, Terminal Server Edition . Windows 2000 Server [1] . Windows 2000 Advanced Server [1] . Windows 2000 Enterprise [1] . Windows 2000 Datacenter [1,2] . Windows XP [2] [1] If terminal services are enabled [2] Probably. I haven't been able to test this The presence of WinFrame or MetaFrame does not affect install mode. If you are installing an application which should apply to (or be used by) all users, you should be in 'install mode' during the installation. The command 'change user /install' enters install mode. 'change user /execute', or logging off the session, exits install mode. The internal effect of install mode is to store registry entries differently. Install mode should be used for system services as well, such as Big Brother. When I installed BBNT on my NT4/TSE machines, I copied the unzipped files from my recovery CD to a directory, used 'change mode /install' to get into install mode, then went through the normal license, install, configure steps you have already documented. Thanks to Paul Bort. Hanging BBNT: ------------- If your of BBNT hangs (set ActivateLog on and check the output of BB.LOG to determine that BBNT stopped running, you shouldn't see any test being executed) then set the ForceRegClose key in HKEY_LOCAL_MACHINE\SOFTWARE\BigBrother\bbnt key to 'N'. Stop/Start BB. If that doesn't work, let us know. CPU and PROCS columns are not generated (or have stopped being generated): -------------------------------------------------------------------------- This is only valid if you get a least the disk status. Make sure that the following key has a value of 0. HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\PerfProc\Performance\Disable Performance Counters If it is set to 1, set it to 0. More info here: http://www.microsoft.com/windows2000/techinfo/reskit/en-us/default.asp?url=/windows2000/techinfo/reskit/en-us/regentry/46705.asp To make sure that this never happens again: http://www.microsoft.com/windows2000/techinfo/reskit/en-us/default.asp?url=/windows2000/techinfo/reskit/en-us/regentry/46705.asp Thanks to Thomas Lauster & Tristan Zondag. WISH LIST: ---------- At install, accept config file (either registry format or proprietary). Make it work with NT clusters (maybe they can be fixed with ForceRegClose)