[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [hobbit] Advice on how to handle HA monitoring

So you are using a custom script to monitor instead of hobbitclient-*.sh ?
This really isn't an option for me, since I have literally dozens of servers that all have the same hobbit homedir, although I have written some custom scripts that check the hostname and only run if they are launched on the host that they need to be on.
I think I will just have the oncall persons manually edit hobbit-clients.cfg in the case of a failover (oncall gets woken up anyhow). They can just uncomment/comment definitions for whichever host is the master.
It would be nice if you could set dependencies for PROC tests, then I could just make all of the PROC tests dependant upon something, like one of the failover daemons, or a flag on the filesystem, etc. 

-Charles

Haertig, David F (Dave) wrote:

I do this with a custom monitoring script (I don't use the standard
Hobbit 'procs' test).

There should be something you can check via script that tells you if a
server is primary or not.  In my case, a database filesystem is mounting
on the primary but not on the secondary.  So my script uses 'df' to look
for that filesystem.  You could use 'mount' as well.  If that database
filesystem is mounted the script does the normal test for processes and
reports red/green.  But if it's not mounted, the script reports a clear
condition. 
-----Original Message-----
From: Charles Jones [mailto:jonescr (at) cisco.com] Sent: Friday, September 21, 2007 1:12 PM 
To: hobbit (at) hswn.dk
Subject: [hobbit] Advice on how to handle HA monitoring

We have 2 hosts, HostA and HostB. They are part of an HA cluster via HP
ServiceGuard. There is a virtual IP and DNS name of "virtual" that
automatically goes to whichever of HostA and HostB is the primary at the
time.

I am currently monitoring both HostA and HostB via Hobbit.  Currently
HostA is the primary, and I am doing various PROC checks. Currently on
HostB, I am not doing process checks.

My problem is, how do I smoothly handle a failover scenario (HostB
becoming the primary)?  When a failover occurs, all of the procs on
HostA are stopped (either by the server crashing, or manualy by
ServiceGuard), and the same procs are started up on HostB.

I'm trying to think of ways to monitor both hosts, but only monitor
procs on the one that is primary. So far the best I can come up with is
to run the hobbit clients in local mode, and maybe have the ServiceGuard
scripts swap out the config files and restart the Hobbit clients when
there is a failover. That would probably work, BUT in this case the
Hobbit homdir is also the same (SAN mount) on both machines, so moving
or editing a file on one does the same on the other :(

Simply shutting down the hobbit client on the non-primary is not an
option, as then it would no longer be monitored at all.

Any ideas? :)

-Charles

To unsubscribe from the hobbit list, send an e-mail to
hobbit-unsubscribe (at) hswn.dk

To unsubscribe from the hobbit list, send an e-mail to
hobbit-unsubscribe (at) hswn.dk