[Xymon] no client results arrive, lot of TIME_WAIT at server

Thu Oct 19 16:42:01 CEST 2017

> On 19 Oct 2017, at 14:49, Norbert Kriegenburg <norbert.kriegenburg at de.ibm.com> wrote:
> 
> Yes,
> that's what i checked at first. But i assume that also the info from the servers own client data are captured by the same client channel listener, and that obviously works.
> 
valid point
> Nothing in the clientdata.log, even if i activate the DEBUG mode with a
> kill -USR2 PID
> 
> My clientdata section:
> # The client back-end module. You need this if you are running the Xymon client on any system.
> [clientdata]
> ENVFILE $XYMONSERVERHOME/etc/xymonserver.cfg
> NEEDS xymond
> CMD /home/xymon/server/bin/xymond_channel --channel=client \
> --log=$XYMONSERVERLOGS/clientdata.log xymond_client \
> --no-clear-files \
> --no-clear-ports \
> --no-port-listing \
> --uptime-status
> INTERVAL 2m
> 
The `INTERVAL` here is not im my and xymon.com’s-version of tasks.cfg— not sure if it does harm (it _may_ try to start a new channel every 2 minutes).
> ps listing:
> $ ps -ef|grep "xymond_channel --channel=client"|grep -v grep
> xymon 18865 18705 0 05:13 ? 00:00:00 /home/xymon/server/bin/xymond_channel --channel=client --log=/home/xymon/server/logs/clientdata.log xymond_client --no-clear-files --no-clear-ports --no-port-listing --uptime-status
> 
If this process does not change it should be fine; your xymon server was restarted 5:13 this morning?
> I have a lot of TIME_WAITS from time to time, no idea why this goes up and down:
> 
> $ netstat -atn|grep 1984
> tcp 0 0 0.0.0.0:1984 0.0.0.0:* LISTEN
> tcp 0 0 10.137.56.43:37752 10.137.56.43:1984 TIME_WAIT
> tcp 0 0 10.137.56.43:37822 10.137.56.43:1984 TIME_WAIT
> ...
> 
> 10.137.56.43 is the Xymon server IP.
> 
> $ netstat -atn|grep 1984|wc -l
> 165
> (sometimes much much more)
> 
> (But goes down to 7 after some mins)
> 
Looks like internal xymon-communication. As long as the TIME_WAIT-count does not go >20k  that should not be the cause.

Thing’s I’d try from here:

1. double-check the XYMSRV / XYMONSERVERS setting on the client(s), e.g. with:

		xymoncmd /usr/bin/env | egrep -i 'xymsrv|xymonservers|bbdisp’

2. just to double-check: tcpdump on xymon server and a client to check for tcp/1984 communication

3. run a channel-listener an the server to see if the client messages (other than the ones from the local client) make it 

		xymoncmd xymond_channel --channel=client cat | grep ‘^client '
		xymoncmd xymond_channel --channel=client cat

	The latter might be very noisy — the former just prints the “header”/1st line of each connection.
	There should be one message for each client sending data every 5 minutes.

HTH
Thomas
> <graycol.gif>Thomas Eckert ---19/10/2017 11:14:56---Norbert, I assume your manual (BB way) is something like `xymon 1.2.3.4 “status example.com.test …”`
> 
> From: Thomas Eckert <thomas.eckert at it-eckert.de>
> To: xymon <xymon at xymon.com>
> Date: 19/10/2017 11:14
> Subject: Re: [Xymon] no client results arrive, lot of TIME_WAIT at server
> Sent by: "Xymon" <xymon-bounces at xymon.com>
> 
> 
> 
> 
> Norbert,
> 
> I assume your manual (BB way) is something like `xymon 1.2.3.4 “status example.com.test …”`.
> 
> The xymon-client send `clientdata` that is processed by a “channel listener” and _that_ sends status-messages similar to above (a bit simplified).
> 
> That channel-listener may not be running/crashing:
> Check our `tasks.cfg` for the section `[clientdata]` and verify, if the `xymond_channel`-process is running (I suspect it’s not). Maybe the `clientdata.log` helps with debugging. You may also run the task manually to get further information.
> 
> What’s your problem with the "lot of TIME_WAIT”? Are these just for tcp/1984-connections?
> 
> Cheers
> Thomas
> On 19 Oct 2017, at 07:30, Norbert Kriegenburg <norbert.kriegenburg at de.ibm.com <mailto:norbert.kriegenburg at de.ibm.com>> wrote:
> experts,
> 
> i have a new (not my first) Xymon installation (4.3.28 on RH 7.4, self compiled).
> All looks good, but the clients only show the network tests (conn, ssh). Only the Xymon server itself shows everything (disk, cpu,...).
> - all clients are RH 7.4, same install image used (i just copied over the client part with client config, create a cronjob for automatic start, so no root access nor extra install needed)
> - nothing suspicious in server or client logs (even in debug mode)
> - no ghosts
> - if i send a test manually (the old BB way) this shows at Xymon immediately (so the client settings are ok)
> - if i shutdown the xymond at server, the client logs complains:
> 
> 2017-10-19 05:09:18.051994 Whoops ! Failed to send message (Connection failed)
> 2017-10-19 05:09:18.052104 -> Could not connect to Xymon daemon at 10.137.56.43 <mailto:daemon at 10.137.56.43>:1984 (Connection refused)
> 
> (so obviously they communicate)
> 
> Querying the xymond shows:
> xymon 127.0.0.1 "query clientname.ssh"
> green <!-- [flags:OrdastLe] --> Thu Oct 19 05:21:29 2017 ssh ok
> is ok, but
> xymon 127.0.0.1 "query clientname.disk"
> shows nothing
> 
> Any ideas?
> 
> Norbert
> 
> _______________________________________________
> Xymon mailing list
> Xymon at xymon.com <mailto:Xymon at xymon.com>
> http://lists.xymon.com/mailman/listinfo/xymon <http://lists.xymon.com/mailman/listinfo/xymon>_______________________________________________
> Xymon mailing list
> Xymon at xymon.com
> http://lists.xymon.com/mailman/listinfo/xymon <http://lists.xymon.com/mailman/listinfo/xymon>
> 
> 
> 
> 
> _______________________________________________
> Xymon mailing list
> Xymon at xymon.com
> http://lists.xymon.com/mailman/listinfo/xymon

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xymon.com/pipermail/xymon/attachments/20171019/3517398d/attachment.html>