xymon client frequently goes purple on guest VMs running on KVM host, then recovers

[Tom Georgoulias]

I'm seeing a problem where the xymon client running on guest VMs on a 
KVM host will go purple for a while, then recover and go back to green. 
  Is anyone else seeing this behavior?  I opened a bug report for this 
(ID: 3057486) but wanted to share it here, just in case someone has 
already identified a fix or a work around.

Here's the info I put in the bug report:

After an undetermined period of time, xymon clients running inside a 
guest VMs stop sending updates to the xymon server, causing purple 
alerts. After approximately 47-48 mins of purple status on the xymon 
server, the xymon client will resume sending regular reports every 5 
mins. This can happen over and over on the same VM, and can occur on 
multiple VMs on the same host server at the same time. The xymon client 
running on the KVM host server does not have this problem.

System Configurations:

1. Guest VMs:
Centos 5 i386
xymon 4.2.3 client (32-bit)
kernel-2.6.18-194.11.1.el5
glibc-common-2.5-49
glibc-2.5-49


2. KVM host server
Centos 5 x86_64
xymon 4.2.3 client (32-bit)
kernel-2.6.18-194.11.1.el5
glibc-2.5-49
glib2-2.12.3-4.el5_3.1
dbus-glib-0.73-8.el5
glibc-common-2.5-49
glibc-2.5-49
avahi-glib-0.6.16-7.el5
glib2-2.12.3-4.el5_3.1
dbus-glib-0.73-8.el5
kvm-83-164.el5_5.15
etherboot-zroms-kvm-5.4.4-13.el5.centos
kmod-kvm-83-164.el5_5.15
kvm-qemu-img-83-164.el5_5.15

Other notes and observations:

1. This problem only occurs on 32-bit guests. None of our 64-bit guests 
experience the problem.
2. The KVM host and guests all use ntpd to keep the clocks in sync and 
ntp is working correctly. The time & date returned by the date command 
is correct while this problem occurs.
3. When this problem occurs, the hobbitlauch binary is still listed in 
the process listing:

ps -ef | grep hobbit

root 5004 4972 0 07:58 pts/0 00:00:00 grep hobbit

hobbit 19345 1 0 Jul27 ? 00:00:10

/home/hobbit/client/bin/hobbitlaunch

--config=/home/hobbit/client/etc/clientlaunch.cfg

--log=/home/hobbit/client/logs/clientlaunch.log

--pidfile=/home/hobbit/client/logs/clientlaunch.<hostname>.pid


Any help is greatly appreciated.

Thanks,
Tom