[hobbit] Advice on how to handle HA monitoring

[Haertig, David F (Dave)]

I do not use custom client scripts on each client machine.  I have a
single custom SERVER script that does it all.  The clients report their
data normally.  My custom server script uses the command...

bb localhost "clientlog clientserver.mydomain.com sections=df,ps" 

...to retrieve the relevent client data, then parses what it finds in
the df section to see if that identifying filesystem is mounted, then
based on that it parses the procs section differently.

-----Original Message-----
From: Charles Jones [mailto:jonescr at cisco.com] 
Sent: Friday, September 21, 2007 2:06 PM
To: hobbit at hswn.dk
Subject: Re: [hobbit] Advice on how to handle HA monitoring

So you are using a custom script to monitor instead of hobbitclient-*.sh
?
This really isn't an option for me, since I have literally dozens of
servers that all have the same hobbit homedir, although I have written
some custom scripts that check the hostname and only run if they are
launched on the host that they need to be on.

I think I will just have the oncall persons manually edit
hobbit-clients.cfg in the case of a failover (oncall gets woken up
anyhow). They can just uncomment/comment definitions for whichever host
is the master.

It would be nice if you could set dependencies for PROC tests, then I
could just make all of the PROC tests dependant upon something, like one
of the failover daemons, or a flag on the filesystem, etc.

-Charles

Haertig, David F (Dave) wrote:
> I do this with a custom monitoring script (I don't use the standard 
> Hobbit 'procs' test).
>
> There should be something you can check via script that tells you if a

> server is primary or not.  In my case, a database filesystem is 
> mounting on the primary but not on the secondary.  So my script uses 
> 'df' to look for that filesystem.  You could use 'mount' as well.  If 
> that database filesystem is mounted the script does the normal test 
> for processes and reports red/green.  But if it's not mounted, the 
> script reports a clear condition.
>
> -----Original Message-----
> From: Charles Jones [mailto:jonescr at cisco.com]
> Sent: Friday, September 21, 2007 1:12 PM
> To: hobbit at hswn.dk
> Subject: [hobbit] Advice on how to handle HA monitoring
>
> We have 2 hosts, HostA and HostB. They are part of an HA cluster via 
> HP ServiceGuard. There is a virtual IP and DNS name of "virtual" that 
> automatically goes to whichever of HostA and HostB is the primary at 
> the time.
>
> I am currently monitoring both HostA and HostB via Hobbit.  Currently 
> HostA is the primary, and I am doing various PROC checks. Currently on

> HostB, I am not doing process checks.
>
> My problem is, how do I smoothly handle a failover scenario (HostB 
> becoming the primary)?  When a failover occurs, all of the procs on 
> HostA are stopped (either by the server crashing, or manualy by 
> ServiceGuard), and the same procs are started up on HostB.
>
> I'm trying to think of ways to monitor both hosts, but only monitor 
> procs on the one that is primary. So far the best I can come up with 
> is to run the hobbit clients in local mode, and maybe have the 
> ServiceGuard scripts swap out the config files and restart the Hobbit 
> clients when there is a failover. That would probably work, BUT in 
> this case the Hobbit homdir is also the same (SAN mount) on both 
> machines, so moving or editing a file on one does the same on the 
> other :(
>
> Simply shutting down the hobbit client on the non-primary is not an 
> option, as then it would no longer be monitored at all.
>
> Any ideas? :)
>
> -Charles
>
> To unsubscribe from the hobbit list, send an e-mail to 
> hobbit-unsubscribe at hswn.dk
>
> To unsubscribe from the hobbit list, send an e-mail to 
> hobbit-unsubscribe at hswn.dk
>   

To unsubscribe from the hobbit list, send an e-mail to
hobbit-unsubscribe at hswn.dk