[hobbit] Highlights of the 4.3.0 version

[Haertig, David F (Dave)]

Sometimes the real world runs interference for Utopia.  While in Utopia
you want to analyse, find the root cause, and fix everything before
proceding, you can't always do that.  When an outage of one hour costs
your company tens of thousands of dollars, you can't justify withholding
a simple bandaid (so long as you don't then ignore the long term fix).

Most everything I do in Hobbit is a custom script.  Restarting crashed
processes is one of the least of my worries.  Although in some rare
cases I do just that (short term), with appropriate logging and email to
the app developement team.  The corporate expense of having the app down
is too great to let Utopian ideas prevail.

Most of the automated Hobbit stuff I do is not restarting dead apps
(luckily, that is very infrequent around here).  It's more mundane.  One
example is disk space.  A full filesystem would shut many things down.
Apps should not fill a filesystem, but sometimes they do.  So my custom
Hobbit scripts first scream and scream about low disk space, even
analysing things down to specific subdirectories and fast growing files
and doing trend analysis.  But if their call is not answered, they start
freeing up space from a "private reserve" I have set aside to deal with
emergencies.  So if we experience a sudden unexpected blowup in a
filesystem at 3am, Hobbit keeps things running in production until the
appropriate people can look into and diagnose the problem.  This may not
be Utopian behavior, but it sure is practical at 3am in the morning!

But my vote would be for Hobbit out-of-the-box to NOT attempt automated
repair actions.  That should be left to the Hobbit administrator.  We
can write custom monitor scripts or custom alert scripts to add this
functionality if it's appropriate for our environments.  It's trivial to
integrate your own scripting into Hobbit.

I sure wish I worked in Utopia though.  The job would be a helluva lot
less stressful!  :-)


-----Original Message-----
From: scottrwalters at gmail.com [mailto:scottrwalters at gmail.com] On Behalf
Of Scott Walters
Sent: Friday, August 03, 2007 11:15 AM
To: hobbit at hswn.dk
Subject: Re: [hobbit] Highlights of the 4.3.0 version

I am definitely in the "monitor only" camp.  As appealing as
"self-healing" may seem, I've seen attempts go horrible wrong too many
times.  For example, shutting down Oracle for upgrades and then being
restarted in the middle of the upgrade.  Not good.

I also agree that "self-healing" lends itself to band-aids that avoid
root-cause determination.  I don't think this requires "baby-sitting,"
but a commitment to fixing things once.  I have also had the displeasure
of making permanent band-aids, but I cannot condone it.

All of those "operational" aspects aside, I've convinced myself from a
security point of view, corrective action from monitoring is bad-- a
clear violation of the separation of duties.  You don't want your
auditors "cleaning up" the numbers as they go over your books.

You know what's better than your webserver being automatically restarted
when it crashes?  Your webserver not crashing.

I completely support the absence of corrective actions from monitor
triggers.  The question I have yet to answer satisfactorily is,"Should
the monitoring system perform additional data collection after specific
errors?"  For example, running a particular "find" command when disk
usage increases to try and identify which files are causing the
partition to fill.


Scott Walters
-PacketPusher

On 8/3/07, Hubbard, Greg L <greg.hubbard at eds.com> wrote:
> Well, I use Netcool which has the opposite philosophy -- there is a 
> "process automation" system that watches processes and restarts them 
> if they fail, while also logging restarts.  You can configure a
"restart"
> parameter to be anything from 0 (forever) to any number of times.  I 
> like to set a reasonable number so persistent errors eventually kill 
> the process, but occasional errors do not.  Log files are not 
> overwritten, but are appended and rotated.
>
> But whatever.  My view seems to be in the minority -- guess the rest 
> of you don't mind 24x7x365 babysitting.
>
> GLH
>
> -----Original Message-----
> From: Galen Johnson [mailto:Galen.Johnson at sas.com]
> Sent: Friday, August 03, 2007 10:18 AM
> To: hobbit at hswn.dk
> Subject: RE: [hobbit] Highlights of the 4.3.0 version
>
> DOn't forget...this is the model that Tivoli and HP Openview, and many

> other commercial monitoring solutions provide and sell as a feature.
> From my experience as a sys admin, I've alwys found that automatically

> restarting a service if it goes down to be "a bad thing"(TM).
>
> In many solutions, logs get overwritten upon a restart that would be 
> integral to the real resolution and prevention.
>
> =G=
>
> -----Original Message-----
> From: Tod Hansmann [mailto:thansmann at directpointe.com]
> Sent: Friday, August 03, 2007 10:40 AM
> To: hobbit at hswn.dk
> Subject: RE: [hobbit] Highlights of the 4.3.0 version
>
> In my experience, I have to agree.  Hobbit is for monitoring so the 
> information that x is down gets to people who can properly diagnose 
> what is going on, not take generic actions.  If generic actions were 
> something that were required for X to function properly, it should be 
> a feature of that software.
>
> Hobbit CAN do some scripting based on alerts, but even that might be a

> bit more than a systems administrator wants to hinder himself with.
>
> Tod Hansmann
> Network Engineer
>
>
> -----Original Message-----
> From: Buchan Milne [mailto:bgmilne at staff.telkomsa.net]
> Sent: Friday, August 03, 2007 12:31 AM
> To: hobbit at hswn.dk
> Cc: Hubbard, Greg L
> Subject: Re: [hobbit] Highlights of the 4.3.0 version
>
> On Tuesday 24 July 2007 22:55:02 Hubbard, Greg L wrote:
> > Wonder if there is any way to tell a client what it's status is so 
> > it can be autonomous?  What I mean is this:  suppose there was a way

> > for the Hobbit client to tell the server that service X was now in 
> > state
> Y,
> > and a client-side module could then activate response Z on its own?
>
> I don't like band-aids like this.
>
> "restart because it's down" prevents the real impact of problems being

> seen, and provides less motivation for fixing things properly. 
> Instead, you sit with frequent short outages (which may avoid the 
> attention of managers, production managers) which have end-user
impact.
>
> I like even less using a monitoring system to do this ...
>
> Regards,
> Buchan
>
> To unsubscribe from the hobbit list, send an e-mail to 
> hobbit-unsubscribe at hswn.dk
>
>
>
>
> To unsubscribe from the hobbit list, send an e-mail to 
> hobbit-unsubscribe at hswn.dk
>
>
>
> To unsubscribe from the hobbit list, send an e-mail to 
> hobbit-unsubscribe at hswn.dk
>
>
>
> To unsubscribe from the hobbit list, send an e-mail to 
> hobbit-unsubscribe at hswn.dk
>
>
>

To unsubscribe from the hobbit list, send an e-mail to
hobbit-unsubscribe at hswn.dk