[Xymon] problem with receiving bbwin data
David Baldwin
david.baldwin at ausport.gov.au
Thu Jul 11 09:18:10 CEST 2013
Phil,
> Thanks again. Yes, snare / winevtmsgs.pl looks like a good work around for this issue....
Let me know if you try it out in anger. I'm not aware of anyone who
actually uses it, and I've probably made some updates. I've also got a
mass of localised rules for event log filtering since the severity
levels, etc can be pretty bogus - errors that can be safely ignored and
"informational" messages that you really need to be alerted on...
David.
> cheers, P
>
>
> ________________________________________
> From: David Baldwin
> Sent: Thursday, 11 July 2013 2:45 PM
> To: Phil Crooker
> Cc: xymon at xymon.com
> Subject: Re: [Xymon] problem with receiving bbwin data
>
> Phil,
>> Thanks, David.... Yes, I don't have clientversion.cfg either - I tried
>> putting in "0.13" in a file of that name in \etc, the client says it
>> reads it and no more error, but who knows what this does or doesn't do...
>>
>> Anyway, on to the rest. As I described in the previous email, I did
>> let the clientlocal.cfg update successfully. But as soon as I add the
>> msgs.dll agent bbwin still ignores the maxdata parameter, tries to
>> send everything and xymond drops the connection before completion
>> (this latest time at 31MB).
>>
>> Do you have this working? Does it actually reduce the size of the msgs
>> sent?
>>
> I have it working, but I don't actually use BBWin for event log
> reporting - I forward all my event logs using Snare to a central syslog
> server and have my own checking against that server-side - see
> http://xymonton.org/monitors:winevtmsgs.pl
>
> My client-local.cfg for Windows clients includes the following to filter
> out everything client-side. You may be able to play with the ignore
> rules to tune it a bit more to your needs while still reducing the size
> of the report. By checking the BBWin\tmp\msg.<clientname>.txt file you
> should see what is taking up all the space.
>
> log:eventlog_security:10240
> ignore .*
> ignore .
> eventlog:security:10240
> ignore handle
> ignore .*
> ignore .
> eventlog:System:10240
> ignore .*
> ignore .
> eventlog:application:10240
> ignore .*
> ignore .
> eventlog:directory service:10240
> ignore .*
> ignore .
>
> David.
>
>> thanks, Phil
>>
>>
>> ------------------------------------------------------------------------
>> *From:* David Baldwin
>> *Sent:* Thursday, 11 July 2013 11:28 AM
>> *To:* Phil Crooker
>> *Cc:* xymon at xymon.com
>> *Subject:* Re: [Xymon] problem with receiving bbwin data
>>
>> Phil,
>>> Thanks for the reply.
>>>
>>> Before continuing, what is supposed to be in logfetch.status? and in
>>> clientversion.cfg?
>>>
>> logfetch.status will possibly only exist if you are checking any
>> regular logfiles - e.g. if client-local.cfg contains something like:
>>
>> [win32]
>> log:c:\WINDOWS\WindowsUpdate.log:20480
>>
>> clientlocal.cfg will contain the relevant section similar to your 4
>> line example below. clientversion.cfg doesn't exist on my systems
>>
>> As BBWin runs, check the size of clientlocal.cfg - if it gets wiped,
>> try saving the contents back again. Also, updating BBWin.cfg should
>> cause BBWin to run immediately without restarting the service. I
>> haven't played around with commenting out msgs.dll
>>
>> David.
>>
>>> So, I have one server with the msgs.dll out of the config. It has
>>> been running for many days with the client updating fine. I redid the
>>> xymon client-local.cfg, stopped bbwin, deleted \tmp\clientlocal.cfg
>>> and restarted it (as it wouldn't update), the \tmp\clientlocal.cfg
>>> now reads what the client-local has:
>>>
>>> eventlog:application:1024
>>> ignore BigBrotherXymonClient
>>> eventlog:system:1025
>>> eventlog:security:10240
>>>
>>> I let that run a cycle, it still updates OK. I then put in the
>>> msgs.dll statement in BBWin.cfg; that registers and BBWin.log shows:
>>>
>>> 2013/07/11 10:42:55 [DEBUG]: [bbwinupdate]: can't open C:\Program
>>> Files (x86)\BBWin\etc\clientversion.cfg : The system cannot find the
>>> file specified.
>>> 2013/07/11 10:42:55 [DEBUG]: [cpu]: cpu review started
>>> 2013/07/11 10:42:55 [DEBUG]: [cpu]: cpu review ended
>>> 2013/07/11 10:42:55 [DEBUG]: [disk]: drive string A:\
>>> 2013/07/11 10:42:55 [DEBUG]: [disk]: drive string C:\
>>> 2013/07/11 10:42:55 [DEBUG]: [disk]: drive string D:\
>>> 2013/07/11 10:42:55 [DEBUG]: [disk]: drive string E:\
>>> 2013/07/11 10:42:55 [DEBUG]: [filesystem]: start
>>> AgentFileSystem::InitCentralMode
>>> 2013/07/11 10:42:55 [DEBUG]: [filesystem]: checking file C:\Program
>>> Files (x86)\BBWin\tmp\clientlocal.cfg
>>> 2013/07/11 10:42:55 [DEBUG]: [filesystem]: reading file C:\Program
>>> Files (x86)\BBWin\tmp\clientlocal.cfg
>>> 2013/07/11 10:42:55 [DEBUG]: [msgs]: start AgentMsgs::InitCentralMode
>>> 2013/07/11 10:42:55 [DEBUG]: [msgs]: checking file C:\Program Files
>>> (x86)\BBWin\tmp\clientlocal.cfg
>>> 2013/07/11 10:42:55 [DEBUG]: [msgs]: reading file C:\Program Files
>>> (x86)\BBWin\tmp\clientlocal.cfg
>>> 2013/07/11 10:42:55 [DEBUG]: [msgs]: will use maxdata size : 1024
>>> 2013/07/11 10:42:55 [DEBUG]: [msgs]: will ignore : BigBrotherXymonClient
>>> 2013/07/11 10:42:55 [DEBUG]: [msgs]: will use maxdata size : 1025
>>> 2013/07/11 10:42:55 [DEBUG]: [msgs]: will use maxdata size : 10240
>>>
>>> and no more updates. The .\tmp\msgs.hostname.txt file is 88MB.
>>>
>>> thanks, Phil
>>>
>>> ------------------------------------------------------------------------
>>> *From:* Scot Kreienkamp
>>> *Sent:* Wednesday, 10 July 2013 10:19 PM
>>> *To:* David Baldwin; Phil Crooker
>>> *Cc:* xymon at xymon.com
>>> *Subject:* RE: [Xymon] problem with receiving bbwin data
>>>
>>>
>>> If the eventlog data being transmitted is too large to transmit until
>>> the clientlocal.cfg is sent back and populated in BBWin, then why not
>>> disable the eventlog client in BBWin until it successfully sends 2-3
>>> status messages? By that time the clientlocal.cfg will be populated
>>> on the client and you can enable the eventlog reader again and
>>> everything should work fine.
>>>
>>>
>>>
>>> *Scot Kreienkamp *
>>>
>>>
>>>
>>> *From:*Xymon [mailto:xymon-bounces at xymon.com] *On Behalf Of *David
>>> Baldwin
>>> *Sent:* Wednesday, July 10, 2013 4:19 AM
>>> *To:* Phil Crooker
>>> *Cc:* xymon at xymon.com
>>> *Subject:* Re: [Xymon] problem with receiving bbwin data
>>>
>>>
>>>
>>> Phil,
>>>
>>> Just found a fix that works for me :)
>>>
>>> I've been having difficulties setting up the bbwin client (ver
>>> 0.13 going to xymon 4.3.10 server) running on win7 sp1, 2008r2
>>> and win2012 boxes in central mode. I've had zero response from
>>> the bbwin-help forum for this, so please bear with me.
>>>
>>>
>>>
>>> For background: I can't get the bbwin client to stop sending all
>>> the logs, it ignores any maxdata parameters I use. Eg:
>>>
>>>
>>>
>>> eventlog:system:1024
>>>
>>>
>>>
>>> It sends everything anyway. If I use log:system:1024, the bbwin
>>> client throws an error that it can't find the system log file. It
>>> does find the file without the maxdata parameter.
>>>
>>>
>>>
>>>
>>> One problem with the settings in client-local.cfg on the xymon server
>>> is that they only propagate to the client after a successful sending
>>> of a client message. The mechanics is that after the client message
>>> has been sent, BBWin does a shutdown on the write socket and then
>>> reads from the read socket until EOF. If the write socket is never
>>> shutdown because the server breaks the connection due to flooding,
>>> then the client-local.cfg section will never be sent to the client.
>>>
>>> I've previously attempted to manually create C:\Program
>>> Files(x86)\BBWin\tmp\clientlocal.cfg to trick it, but with limited
>>> success. Just played around with it again - with client reporting to
>>> my primary server I just got it going - previously I was reporting to
>>> 2 servers. The trick seems to be in having logfetch.status also
>>> created. I then modified my BBWin.cfg back to 2 servers and it broke
>>> again. Maybe the 2nd server is responding differently. It is
>>> undefined which server's clientlocal.cfg to believe anyway :) Swapped
>>> the order of the servers around and it seems stable.
>>>
>>> If the server just discarded all the flooding data beyond MAXMSG_*
>>> and then returned the client config section anyway maybe it could be
>>> made to work, but there may be good reasons why that would not be
>>> sensible.
>>>
>>> This aside, I do need to get this working as I have to start
>>> monitoring 10 new windows servers and have to monitor the event logs,
>>> so I can't just stop sending them as has been suggested. Yes I could
>>> do some powershell scripts and send them via bbwincmd but the bbwin
>>> client is made for this task.....
>>>
>>>
>>>
>>> So, looking at this from the other side, the xymon server appears to
>>> be resetting the session after about 22MB of data has been sent (I
>>> know, this is ludicrous, but it is windows). Nothing in the xymond
>>> logs (except for the occasional data flooding error ("1st line
>>> client", always)); on the client side it reports it can't send the
>>> data to the xymon server.
>>>
>>>
>>>
>>> Depending on what auditting you have enabled, 22MB is very easy to
>>> achieve! Turn on Security event logging including success and failure
>>> and it's almost guaranteed :)
>>>
>>> David.
>>>
>>> I've set the MAXMSG_* to quite silly levels:
>>>
>>>
>>>
>>> # ipcs
>>>
>>> ------ Shared Memory Segments --------
>>>
>>> key shmid owner perms bytes
>>> nattch status
>>>
>>> 0x01034be7 16908288 xymon 600 102400000 2
>>>
>>>
>>> 0x02034be7 16941057 xymon 600 102400000 2
>>>
>>>
>>> 0x03034be7 16973826 xymon 600 102400000 2
>>>
>>>
>>> 0x04034be7 17006595 xymon 600 102400000 2
>>>
>>>
>>> 0x05034be7 17039364 xymon 600 262144 1
>>>
>>>
>>> 0x06034be7 17072133 xymon 600 32768 1
>>>
>>>
>>> 0x07034be7 17104902 xymon 600 102400000 2
>>>
>>>
>>> 0x08034be7 17137671 xymon 600 102400000 2
>>>
>>>
>>> 0x09034be7 17170440 xymon 600 131072 1
>>>
>>>
>>>
>>>
>>> Anything up to and including this size has no effect on the problem.
>>>
>>>
>>>
>>> Looking at the tcpdump stream, the bbwin client sends data normally
>>> with regular ACKs from xymond till around that 22MB mark then xymond
>>> responds with a FIN packet, then with RST packets and the session
>>> shuts down. Nothing in the packets themselves indicate what the
>>> problem is.
>>>
>>>
>>>
>>> If anyone can help with this, please, it would be great.
>>>
>>>
>>>
>>> thanks, Phil
>>>
>>>
>>>
> --
> David Baldwin - Senior Systems Administrator (Datacentres + Networks)
> Information and Communication Technology Services
> Australian Sports Commission http://ausport.gov.au
> Tel 02 62147830 Fax 02 62141830 PO Box 176 Belconnen ACT 2616
> david.baldwin at ausport.gov.au Leverrier Street Bruce ACT 2617
>
>
> -------------------------------------------------------------------------------------
> Keep up to date with what's happening in Australian sport visit http://www.ausport.gov.au
>
> This message is intended for the addressee named and may contain confidential and privileged information. If you are not the intended recipient please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. If you receive this message in error, please delete it and notify the sender.
> -------------------------------------------------------------------------------------
>
> ________________________________
> Please consider the environment before printing this e-mail
>
> This message from ORIX Australia may contain confidential and/or privileged information. If you are not the intended recipient, any use, disclosure or copying of this message (or of any attachments to it) is not authorised. If you have received this message in error, please notify the sender immediately and delete the message and any attachments from your system. Please inform the sender if you do not wish to receive further communications by email. ORIX handles personal information according to a Privacy Policy that is consistent with the National Privacy Principles. Please let us know if you would like a copy.
>
> It is also available at www.orix.com.au<http://www.orix.com.au>
>
--
David Baldwin - Senior Systems Administrator (Datacentres + Networks)
Information and Communication Technology Services
Australian Sports Commission http://ausport.gov.au
Tel 02 62147830 Fax 02 62141830 PO Box 176 Belconnen ACT 2616
david.baldwin at ausport.gov.au Leverrier Street Bruce ACT 2617
More information about the Xymon
mailing list