[hobbit] hobbitfetch replacement with ssh

Tue Jun 26 05:26:11 CEST 2007

p.s. - I posted these scripts a monent ago.  I see in the email replay
that my cut-n-paste did not work 100%.  It appears that everything is
there, but some newlines were lost in a few places.  e.g., you will see:

${BBCMD} --env=${ENVFILE} ${CMD} >${LOGFILE} 2>&1 exit 0

... but if you know shell scripts, you'll recognize that the "exit 0"
part should be on a line by itself.  I do not know why cut-n-paste
didn't work right.  I guess some Linux/Solaris/Windows/Thunderbird
glitch.  There are a few other places where commands were run-on like
the above example.  Watch for them, and manually fix them.

Sorry about that!

-----Original Message-----
From: Haertig, David F (Dave) 
Sent: Monday, June 25, 2007 9:20 PM
To: 'hobbit at hswn.dk'
Subject: RE: [hobbit] hobbitfetch replacement with ssh

Below is what I ended up doing.  It's working like a champ.  It probably
doesn't fit your environment exactly, but it may be of some help anyway.

My Hobbit server is Linux 2.6, the clients I'm pulling data from are
Solaris 9.  All running OpenSSH.  The client domains are picked up on
the Hobbit server via /etc/resolv.conf mechanism.  Hobbit server has
socks client installed and configured for our environment.  ssh public
keys were distributed from host to clients so ssh/scp will work without
passwords being required.

The file "hobbitclient-custom.sh" you see below is nothing more than the
standard hobbitclient.sh file with some stuff I didn't want stripped
out.  I modified the LOGFETCHCFG line to point to the "scripts"
directory rather than the default "tmp" directory because I wanted to
keep all my customized stuff in that scripts directory.

Custom HOST script, "pulldata.sh", found on server in $BBHOME/ext
directory, run via clientlaunch.cfg
========================================================================
=============================
export SOCKS5_USER='my_socks_userid'
export SOCKS5_PASSWD='my_socks_password'
USER=stars
for DMZ_MACH in 'machine1' 'machine2' 'machine3'
do
  TMPFILE=${BBTMP}/pulldata.${DMZ_MACH}.tmp.$$
  runsocks ssh ${USER}@${DMZ_MACH} hobbit/scripts/exec_client.sh
  runsocks scp
${USER}@${DMZ_MACH}:~${USER}/hobbit/tmp/msg.${DMZ_MACH}.txt ${TMPFILE}
  ${BB} ${BBDISP} "`cat ${TMPFILE}`"
  rm -f ${TMPFILE}
done

Custom CLIENT script, "exec_client.sh", found on client in
/home/stars/hobbit/scripts directory
========================================================================
=======================
#!/bin/sh
BBHOME=/home/stars/hobbit
BBCMD=${BBHOME}/bin/bbcmd
CMD=${BBHOME}/scripts/hobbitclient-custom.sh
ENVFILE=${BBHOME}/etc/hobbitclient.cfg
LOGFILE=${BBHOME}/logs/hobbitclient-custom.log
${BBCMD} --env=${ENVFILE} ${CMD} >${LOGFILE} 2>&1 exit 0

Custom CLIENT script, "hobbitclient-custom.sh", found on client in
/home/stars/hobbit/scripts directory
========================================================================
===============================
#!/bin/sh
LANG=C
LC_ALL=C
LC_MESSAGES=C
export LANG LC_ALL LC_MESSAGES
if test "$BBOSSCRIPT" = ""; then
  BBOSSCRIPT="hobbitclient-`uname -s | tr '[A-Z]' '[a-z]'`.sh"
fi
MSGFILE="$BBTMP/msg.$MACHINEDOTS.txt"
MSGTMPFILE="$MSGFILE.$$"
LOGFETCHCFG=$BBHOME/scripts/logfetch.$MACHINEDOTS.cfg
LOGFETCHSTATUS=$BBTMP/logfetch.$MACHINEDOTS.status
export LOGFETCHCFG LOGFETCHSTATUS
rm -f $MSGTMPFILE
touch $MSGTMPFILE
echo "client $MACHINE.$BBOSTYPE $CONFIGCLASS" >>$MSGTMPFILE
$BBHOME/bin/$BBOSSCRIPT >> $MSGTMPFILE if test -f $LOGFETCHCFG then
  $BBHOME/bin/logfetch $LOGFETCHCFG $LOGFETCHSTATUS >>$MSGTMPFILE fi
echo "[clock]" >>$MSGTMPFILE $BBHOME/bin/logfetch --clock >>$MSGTMPFILE
rm -f $MSGFILE mv $MSGTMPFILE $MSGFILE exit 0

Custom CLIENT file, "logfetch.machine1.txt", found on client in
/home/stars/hobbit/scripts directory
========================================================================
============================
log:/my_app/logs/file1:4096
log:/my_app/logs/file2:4096
log:/my_app/logs/file3:4096

-----Original Message-----
From: John G [mailto:jg2727 at gmail.com]
Sent: Monday, June 25, 2007 8:30 PM
To: hobbit at hswn.dk
Subject: Re: [hobbit] hobbitfetch replacement with ssh

Here are some old messages on the topic that are similar to what you are
doing.

http://www.hswn.dk/hobbiton/2006/01/msg00045.html
http://www.hswn.dk/hobbiton/2006/01/msg00047.html

On 6/23/07, Stef Coene <stef.coene at docum.org> wrote:
> On Saturday 23 June 2007, Haertig, David F (Dave) wrote:
> > Using bbcmd to run hobbitclient.sh was a good idea, but presented 
> > some problems of it's own.  No problem getting it to set the 
> > environment though.
> >
> > For one, invoking the client in this manner leaves lots of 
> > iostatcpu, iostatdisk, and vmstat files laying around in the tmp 
> > directory.  I guess it must be hobbitlaunch that cleans these up
normally.
> >
> > Second, I could not trick logfetch into grabbing custom logfiles.  I

> > tried putting the appropriate entries in ..../tmp/logfetch.XXX.cfg 
> > on the client end, but logfetch ignored these.  This file is 
> > normally overwritten each time the client contacts the host, so 
> > normally you wouldn't edit it.  But in my case the client is not 
> > connecting to the host so I thought it might be "safe" to manually 
> > create this file, but alas logfetch appears to ignore it.
> >
> > I may well run into this logfetch gotcha no matter how I invoke 
> > hobbitclient.sh  If the client cannot talk to the Hobbit server, as 
> > in my DMZ case, is it even possible to configure logfetch to grab 
> > custom logfiles?  The logfetch config is normally downloaded from
the server.
> > I don't see any way (yet) to configure it locally on the client.
> If "I" should have this problem, I will try this:
> change hobbit client.sh so it:
> - is not trying to contact the hobbit server (so the bb command is not
> used)
> - the message file is saved in a fixed file (so the pid is not 
> appended to it)
> - the logfetch can still work if you update the input file manually
>
> make a daemon on the hobbit server:
> - that runs every minute
> - contacts the hobbit client with ssh and checks for a message file, 
> if a message file is found, the content is send to the hobbit server 
> and the file is deleted
>
> Or something like this ;)
>
>
> Stef
>
> To unsubscribe from the hobbit list, send an e-mail to 
> hobbit-unsubscribe at hswn.dk
>
>
>

To unsubscribe from the hobbit list, send an e-mail to
hobbit-unsubscribe at hswn.dk