[hobbit] Restarting failed processes on the client

[Henrik Stoerner]

On Wed, Jul 11, 2007 at 04:13:56PM +0200, Henrik Stoerner wrote:
> If You really want this, then the easiest way is probably to
> have a script on the Hobbit server that handles the service
> restart, and trigger it from an alerting script. Here's how:

[snipped]

Particularly for ssh, running the recovery script from the Hobbit
server might not be easy - since ssh is usually the only way you 
can remote-login to the server and gets things (re-)started.

So to implement the same functionality on the client-side, you can
write a client-side extension script that does:

   #!/bin/sh

   PROCSTATUS=`$BB $BBDISP "query $MACHINE.procs" | awk '{print $1}'`
   if test "$PROCSTATUS" = "red"
   then
      /etc/init.d/sshd restart
   fi

   exit 0

This triggers the "sshd restart" whenever the "procs" status goes red.
So it won't be able to tell if it's the sshd process that triggers a red
if you're monitoring multiple processes on each host. So alternatively,
you could add network-monitoring of "ssh", and then query the "ssh"
column instead of the "procs" column.


Regards,
Henrik