[hobbit] Advice on how to handle HA monitoring

[Asif Iqbal]

On 9/21/07, Charles Jones <jonescr at cisco.com> wrote:
> So you are using a custom script to monitor instead of hobbitclient-*.sh ?
> This really isn't an option for me, since I have literally dozens of
> servers that all have the same hobbit homedir, although I have written

It does not matter. You can still write an external script like this

[external]
        ENVFILE /path/to/local/etc/hobbitclient.cfg
        CMD /path/to/local/external/script
        LOGFILE /path/to/local/logs/external.log
        INTERVAL 5m

That will work just fine even though you are sharing same hobbit client

> some custom scripts that check the hostname and only run if they are
> launched on the host that they need to be on.
>
> I think I will just have the oncall persons manually edit
> hobbit-clients.cfg in the case of a failover (oncall gets woken up
> anyhow). They can just uncomment/comment definitions for whichever host
> is the master.

Ouch!

>
> It would be nice if you could set dependencies for PROC tests, then I
> could just make all of the PROC tests dependant upon something, like one
> of the failover daemons, or a flag on the filesystem, etc.
>
> -Charles
>
> Haertig, David F (Dave) wrote:
> > I do this with a custom monitoring script (I don't use the standard
> > Hobbit 'procs' test).
> >
> > There should be something you can check via script that tells you if a
> > server is primary or not.  In my case, a database filesystem is mounting
> > on the primary but not on the secondary.  So my script uses 'df' to look
> > for that filesystem.  You could use 'mount' as well.  If that database
> > filesystem is mounted the script does the normal test for processes and
> > reports red/green.  But if it's not mounted, the script reports a clear
> > condition.
> >
> > -----Original Message-----
> > From: Charles Jones [mailto:jonescr at cisco.com]
> > Sent: Friday, September 21, 2007 1:12 PM
> > To: hobbit at hswn.dk
> > Subject: [hobbit] Advice on how to handle HA monitoring
> >
> > We have 2 hosts, HostA and HostB. They are part of an HA cluster via HP
> > ServiceGuard. There is a virtual IP and DNS name of "virtual" that
> > automatically goes to whichever of HostA and HostB is the primary at the
> > time.
> >
> > I am currently monitoring both HostA and HostB via Hobbit.  Currently
> > HostA is the primary, and I am doing various PROC checks. Currently on
> > HostB, I am not doing process checks.
> >
> > My problem is, how do I smoothly handle a failover scenario (HostB
> > becoming the primary)?  When a failover occurs, all of the procs on
> > HostA are stopped (either by the server crashing, or manualy by
> > ServiceGuard), and the same procs are started up on HostB.
> >
> > I'm trying to think of ways to monitor both hosts, but only monitor
> > procs on the one that is primary. So far the best I can come up with is
> > to run the hobbit clients in local mode, and maybe have the ServiceGuard
> > scripts swap out the config files and restart the Hobbit clients when
> > there is a failover. That would probably work, BUT in this case the
> > Hobbit homdir is also the same (SAN mount) on both machines, so moving
> > or editing a file on one does the same on the other :(
> >
> > Simply shutting down the hobbit client on the non-primary is not an
> > option, as then it would no longer be monitored at all.
> >
> > Any ideas? :)
> >
> > -Charles
> >
> > To unsubscribe from the hobbit list, send an e-mail to
> > hobbit-unsubscribe at hswn.dk
> >
> > To unsubscribe from the hobbit list, send an e-mail to
> > hobbit-unsubscribe at hswn.dk
> >
>
> To unsubscribe from the hobbit list, send an e-mail to
> hobbit-unsubscribe at hswn.dk
>
>
>


-- 
Asif Iqbal
PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu