[hobbit] Advice on how to handle HA monitoring

[Charles Jones]

So you are using a custom script to monitor instead of hobbitclient-*.sh ?
This really isn't an option for me, since I have literally dozens of 
servers that all have the same hobbit homedir, although I have written 
some custom scripts that check the hostname and only run if they are 
launched on the host that they need to be on.

I think I will just have the oncall persons manually edit 
hobbit-clients.cfg in the case of a failover (oncall gets woken up 
anyhow). They can just uncomment/comment definitions for whichever host 
is the master.

It would be nice if you could set dependencies for PROC tests, then I 
could just make all of the PROC tests dependant upon something, like one 
of the failover daemons, or a flag on the filesystem, etc.

-Charles

Haertig, David F (Dave) wrote:
> I do this with a custom monitoring script (I don't use the standard
> Hobbit 'procs' test).
>
> There should be something you can check via script that tells you if a
> server is primary or not.  In my case, a database filesystem is mounting
> on the primary but not on the secondary.  So my script uses 'df' to look
> for that filesystem.  You could use 'mount' as well.  If that database
> filesystem is mounted the script does the normal test for processes and
> reports red/green.  But if it's not mounted, the script reports a clear
> condition. 
>
> -----Original Message-----
> From: Charles Jones [mailto:jonescr at cisco.com] 
> Sent: Friday, September 21, 2007 1:12 PM
> To: hobbit at hswn.dk
> Subject: [hobbit] Advice on how to handle HA monitoring
>
> We have 2 hosts, HostA and HostB. They are part of an HA cluster via HP
> ServiceGuard. There is a virtual IP and DNS name of "virtual" that
> automatically goes to whichever of HostA and HostB is the primary at the
> time.
>
> I am currently monitoring both HostA and HostB via Hobbit.  Currently
> HostA is the primary, and I am doing various PROC checks. Currently on
> HostB, I am not doing process checks.
>
> My problem is, how do I smoothly handle a failover scenario (HostB
> becoming the primary)?  When a failover occurs, all of the procs on
> HostA are stopped (either by the server crashing, or manualy by
> ServiceGuard), and the same procs are started up on HostB.
>
> I'm trying to think of ways to monitor both hosts, but only monitor
> procs on the one that is primary. So far the best I can come up with is
> to run the hobbit clients in local mode, and maybe have the ServiceGuard
> scripts swap out the config files and restart the Hobbit clients when
> there is a failover. That would probably work, BUT in this case the
> Hobbit homdir is also the same (SAN mount) on both machines, so moving
> or editing a file on one does the same on the other :(
>
> Simply shutting down the hobbit client on the non-primary is not an
> option, as then it would no longer be monitored at all.
>
> Any ideas? :)
>
> -Charles
>
> To unsubscribe from the hobbit list, send an e-mail to
> hobbit-unsubscribe at hswn.dk
>
> To unsubscribe from the hobbit list, send an e-mail to
> hobbit-unsubscribe at hswn.dk
>