[hobbit] Re: Status not available

[Brian Lynch]

Looks like 'larrddata'  is clean.  So far, only 'larrdstatus' is
causing it to crash.

- Brian


On Tue, 1 Mar 2005 10:48:01 -0800, Brian Lynch <brianlynch at gmail.com> wrote:
> OK... I enabled the modules one at a time until I reached
> 'larrdstatus'.   When I enabled that module, hobbitd began crashing
> again.  Here is the contents of the hobbitlaunch.cfg.
> 
> #
> # The hobbittasks.cfg file is loaded by "hobbitlaunch".
> # It controls which of the Hobbit modules to run, how often, and #
> with which parameters, options and environment variables. #
> 
> # This is the main Hobbit daemon. You cannot live without this one. [hobbitd]
>         HEARTBEAT
>         ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg
>         CMD hobbitd --debug --restart=$BBTMP/hobbitd.chk
> --checkpoint-file=$BBTMP/hobbitd.chk --checkpoint-interval=600
> --log=$BBSERVERLOGS/hobbitd.log --admin-senders=127.0.0.1,$BBSERVERIP
> 
> # "bbstatus" saves status-logs in text- and html-format, like the old
> Big Brother
> # daemon does. Unless you are using add-ons that directly access the
> log-files, you
> # will not need to run this module, and it is recommended that you
> keep it disabled # since storing the raw logs on disk can cause a
> significant load on your server.
> 
> [bbstatus]
>         DISABLED
>         ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg
>         NEEDS hobbitd
>         CMD hobbitd_channel --channel=status --log=$BBSERVERLOGS/status.log
> hobbitd_filestore --status --html
> 
> # "bbhistory" keeps track of the status changes that happen, in a
> manner that is # compatible with the Big Brother history logs. You
> probably do want to run this.
> 
> [bbhistory]
>         ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg
>         NEEDS hobbitd
>         CMD hobbitd_channel --channel=stachg --log=$BBSERVERLOGS/history.log
> hobbitd_history
> 
> # "bbdata" saves information sent using the BB "data" protocol, like
> the old Big Brother
> # daemon does. Unless you are using add-ons that directly access the
> data-files, you
> # will not need to run this module, and it is recommended that you
> keep it disabled # since storing the raw data on disk can cause a
> significant load on your BB server. # LARRD uses data-files, but since
> LARRD is handled by the hobbitd_larrd module, you do # not need to run
> the "bbdata" module to get LARRD graphs.
> 
> [bbdata]
>         DISABLED
>         ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg
>         NEEDS hobbitd
>         CMD hobbitd_channel --channel=data   --log=$BBSERVERLOGS/data.log
> hobbitd_filestore --data
> 
> # "bbnotes" saves web note-files, that are sent using the BB "notes"
> protocol. This is # disabled by default; if you use the BB "notes"
> protocol, then you should enable this.
> 
> [bbnotes]
>         DISABLED
>         ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg
>         NEEDS hobbitd
>         CMD hobbitd_channel --channel=notes  --log=$BBSERVERLOGS/notes.log
> hobbitd_filestore --notes
> 
> # "bbenadis" updates the files used to indicate that a host or test
> has been enabled or disabled. # These files are used by bbgen and the
> "maint.pl" script to determine what is currently enabled # and
> disabled, so you probably want to run this module.
> 
> [bbenadis]
>         ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg
>         NEEDS hobbitd
>         CMD hobbitd_channel --channel=enadis --log=$BBSERVERLOGS/enadis.log
> hobbitd_filestore --enadis
> 
> # "bbpage" sends out alerts. Note that this module is NOT compatible
> with the old Big Brother # system - it uses a different configuration
> file to determine how alerts get sent. If you want # alerts to go out
> via pager, e-mail or some other means, then you must run this module.
> 
> [bbpage]
>         ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg
>         NEEDS hobbitd
>         CMD hobbitd_channel --channel=page   --log=$BBSERVERLOGS/page.log hobbitd_alert
> 
> # "larrdstatus" updates RRD files with information that arrives as
> "status" messages. # If you want RRD graphs of your monitoring data,
> then you want to run this.
> 
> [larrdstatus]
>         ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg
>         NEEDS hobbitd
>         CMD hobbitd_channel --channel=status
> --log=$BBSERVERLOGS/larrd-status.log hobbitd_larrd --rrddir=$BBVAR/rrd
> 
> # "larrddata" updates RRD files with information that arrives as
> "data" messages. # If you want RRD graphs of your monitoring BB data,
> then you want to run this.
> 
> [larrddata]
>         DISABLED
>         ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg
>         NEEDS hobbitd
>         CMD hobbitd_channel --channel=data
> --log=$BBSERVERLOGS/larrd-data.log hobbitd_larrd --rrddir=$BBVAR/rrd
> 
> # "bbdisplay" runs the bbgen tool to generate the Hobbit webpages from
> the status information that # has been received. Big Brother updated
> the webpages once every 5 minutes. The default here is to # run it
> every minute for faster updates, but you can change it if you have a
> highly loaded server # and dont need updates that often.
> 
> [bbdisplay]
>         ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg
>         NEEDS hobbitd
>         GROUP generators
>         CMD bb-display.sh
>         LOGFILE $BBSERVERLOGS/bb-display.log
>         INTERVAL 1m
> 
> # "larrdcolumn" is responsible for updating the contents of the LARRD
> overview page, found on the # "trends" column for each host. Since the
> set of graphs does not change very often, we run this # less
> frequently than the normal webpage updates. We also make sure (with
> the "GROUP" setting) # that they don't run simultaneously with the
> infocolumnn and bbdisplay tasks. [larrdcolumn]
>         DISABLED
>         ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg
>         NEEDS hobbitd
>         GROUP generators
>         CMD bb-larrdcolumn --hobbitd --rrddir=$BBVAR/rrd --column=trends
>         LOGFILE $BBSERVERLOGS/bb-display.log
>         INTERVAL 15m
> 
> # "infocolumn" is responsible for updating the contents of the INFO
> pages, found on the # "info" column for each host. Since the content
> does not change unless there is a
> # configuration change, we update these less frequently than the
> normal webpage updates. # We also make sure (with the "GROUP" setting)
> that they don't run simultaneously with
> # the larrdcolumnn and bbdisplay tasks.
> [infocolumn]
>         DISABLED
>         ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg
>         NEEDS hobbitd
>         GROUP generators
>         CMD bb-infocolumn --hobbitd --column=info
>         LOGFILE $BBSERVERLOGS/bb-display.log
>         INTERVAL 15m
> 
> # "bbnet" runs the bbtest-net tool to perform the network based tests
> - i.e. http, smtp, ssh, dns and # all of the various network protocols
> we need to test.
> 
> [bbnet]
>         ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg
>         NEEDS hobbitd
>         CMD bbtest-net --report --ping --checkresponse
>         LOGFILE $BBSERVERLOGS/bb-network.log
>         INTERVAL 5m
> 
> # "bbretest" picks up the tests that the normal network test consider
> "failed", and re-does those # tests more often. This enables Big
> Brother to pick up a recovered network service faster than # if it
> were tested only by the "bbnet" task (which only runs every 5
> minutes). So if you have # servers with very high availability
> guarantees, running this task will make your availability # reports
> look much better.
> 
> [bbretest]
>         ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg
>         NEEDS hobbitd
>         CMD $BBHOME/ext/bbretest-net.sh
>         LOGFILE $BBSERVERLOGS/bb-retest.log
>         INTERVAL 1m
> 
> 
> On Tue, 1 Mar 2005 10:10:12 -0800, Brian Lynch <brianlynch at gmail.com> wrote:
> > Henrik,
> >
> > Tried the patch and hobbitd still crashed.
> >
> > Tried the disable module test.  Disabled all the modules you mentioned
> > above and hobbitd has not crashed since 8:30 PST this morning.  I'll
> > try adding back in modules until it crashes.
> >
> > - Brian
> >
> > On Tue, 1 Mar 2005 09:33:51 +0100, Henrik Stoerner <henrik at hswn.dk> wrote:
> > > I've found something that might explain it, but I am certainly not
> > > sure if it's the cause of your problems. This bug would only trigger
> > > if some client of yours was sending in very large "data" messages,
> > > like 100 KB or more.
> > >
> > > Could you try the attached patch and let me know if it solves your
> > > problem?
> > >
> > > If not, I would like you to try and disable all of the hobbitd worker
> > > modules in hobbitlaunch.cfg - i.e. bbstatus, bbhistory, bbdata,
> > > bbnotes, bbenadies, bbpage, larrdstatus and larrddata. Restart hobbit
> > > and see if it still crashes. If it doesn't then we've narrowed down
> > > the problem - if it does, then the problem is somewhere else than
> > > where I've been looking so far.
> > >
> > > Thanks,
> > > Henrik
> > >
> > >
> > > To unsubscribe from the hobbit list, send an e-mail to
> > > hobbit-unsubscribe at hswn.dk
> > >
> > >
> > >
> > >
> >
>