[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [hobbit] Http login test

On Mon, May 25, 2009 at 10:23 AM, Ralph Mitchell <ralphmitchell (at) gmail.com>wrote:

> It was titled "Hobbit content check script".  Here's the body of the
> message, and the attachments:
>
> Sorry I didn't get back to you earlier - I've been fighting with perl
> & SOAP, trying to get some status messages out of a bunch of load
> balancers.  I think I finally got it now....
>
> Attached (hopefully :) are a couple of files I tried to post to the
> list some time last year.  There's a shell script that does a login to
> a Siteminder site, and a perl script evolved from something Daniel
> Stenberg wrote - he's the original author of curl.  Daniel's very
> active on the curl mailing list, by the way, along with a bunch of
> other helpful folks, if it starts going weird on you.
>
> The Siteminder login probably won't be much use to you as is, but it
> demonstrates a few things I've worked out over the years.  Like, the
> standard options string to pass to curl, collecting messages along the
> way, and loading up the message string with a Manual Check url at the
> end for ops to try the check for themselves before waking people up...
>
> You can put almost *anything* into the message string that gets sent
> to Hobbit.  Right now I have scripts that check car rentals at travel
> sites for a particular company and emit a table of the current prices.
> Another script hits a cruise monitor and puts out a list using the
> &red and &green dots to mark cruise lines as reachable or not.  You
> can push out font color and size changes, urls to saved pages, etc.
>
> As far as cookies go, I generally just have curl save them whenever it
> finishes:
>
>  curl -b cookies -c cookies .....
>
> If you do that you can go look at the cookies and see what came in.
> Other things become possible too, such as editing cookies in between
> curl calls.  If you do keep cookies, though, the *first* thing you
> should do in the script is delete the cookie file.  That's like
> clearing the browser cookie cache, and can avoid a lot of mysterious
> errors.
>
> Finally, if you think you might spend some time on web site scripts,
> there's a Firefox extension that is absolutely invaluable -
> LiveHTTPheaders.  It shows you *exactly* what the browser sends to the
> server, as well as what it gets back - not the page content, just the
> headers, but that's all you need to recreate the sequence.  It even
> works twith secure sites.  If you see something going out that you
> can't account for, the usual suspect is javascript.  I've seen
> javascript that assembles a url on the fly from form elements, fiddles
> with some of the form elements, then posts the form.  That kind of
> thing can be difficult to nail down, and almost impossible to
> replicate...
>
> Ralph Mitchell
>

The attachments may not have gone through, so here they are again:

==========  bb-siteminder.sh ==========
#!/bin/sh

# In case of disaster, uncomment the next line and watch the output
# set -x
#
# This script runs out of cron.  Tell it where to find Big Brother.
BBHOME=/home/bb/bb18b3
export BBHOME

TESTHOST=system.domain.com
URL="http://$TESTHOST/";
ORIGURL="$URL"
USERNAME=bbusername
PASSWORD=bbsecretword

# Where to stash the downloaded html files.  These get patched into the BB
# report so that the monitoring guys can click through and see what we got.
BBDIR="$BBHOME/www/siteminder"

# Gonna need a cookie jar.  Start off empty, so each checkout starts clean.
COOKIES="$BBDIR/$TESTHOST.cookies"
rm -f $COOKIES

# Some standard curl flags.  See the curl man page for details
STDOPTS="-w %{url_effective} -L --connect-timeout 60 --max-time 120 -s -S -b
$COOKIES -c $COOKIES -A Mozilla/4.79"

# This is just a tag for identifying environment dumps.
BBPROG=bb-siteminder.sh; export BBPROG

# The next block came from the BB ext example script.  If BBHOME is not set
# there's no point continuing.  If BBTMP is not set, load up the standard
# environment from bbdef.sh.
if test "$BBHOME" = ""
then
        echo "BBHOME is not set... exiting"
        exit 1
fi

if test ! "$BBTMP"                      # GET DEFINITIONS IF NEEDED
then
         # echo "*** LOADING BBDEF ***"
        . $BBHOME/etc/bbdef.sh          # INCLUDE STANDARD DEFINITIONS
fi

# This is used to label the result column
TEST="login"

# Make sure bail out code is set.
bailout=0

# ========== step 1 ==========
# Grab the first page
STEP=step1
# Name the file for the host, test and step, to make it unique
FILENAME=$BBDIR/$TESTHOST.$TEST.$STEP.html
# remove any previous copy to avoid misunderstandings
rm -f $FILENAME

# Grab the first page.  The effective url is recorded for patching
# into the action url is necessary
effect=`$CURL $STDOPTS \
  -o $FILENAME \
  $URL`
# Save the return code, it might be important
ret=$?

# code 52 means "the server sent nothing"
if [ "$ret" -eq "52" ]; then
  # empty reply, try it again
  MESSAGE="$STEP: failed with curl error code: 52 - Empty reply from server.
 Url: <a href=$URL>$URL</a>"
  MESSAGE="$MESSAGE
$STEP: retrying..."
  sleep 2
  effect=`$CURL $STDOPTS \
    -o /dev/null \
    $URL`
  ret=$?
fi

if [ -s $FILENAME ]; then
  # file exists and has size greater than zero.  look for something to
validate
  TESTSTR="Something from the Login Page"
  check=`$GREP -ic "$TESTSTR" $FILENAME`
  if [ "$check" -gt "0" ]; then
    # Found the test string, we're good to continue
    MESSAGE="$MESSAGE
$STEP: Got Login page - found <b>$TESTSTR</b>"
  else
    # Didn't find the test string, bail out of subsequent steps
    MESSAGE="$MESSAGE
$STEP: failed to find <b>$TESTSTR</b>.  curl error code: $ret.  Url: <a
href=$URL>$URL</a>"
    bailout=1
  fi
else
  # Got nothing useful from the server, give up
  MESSAGE-"$MESSAGE
$STEP: failed with empty file.  curl error code: $ret.  Url: <a
href=$URL>$URL</a>"
  bailout=1
fi

# If previous step completed OK, perform next step
if [ "$bailout" -eq "0" ]; then
  # formextract.pl strips out the form elements and formats them like this:
  #   -d username=
  #   -d password=
  #   http://server.domain.com/path/to/form/processor
  # On some pages, the action url is not fully qualified.
  FORM=`cat $FILENAME | $HOME/bin/formextract.pl`
  # Pull out the form variable lines and patch in the userid & password
  FORMVARS=`echo "$FORM" | $GREP -- '-d' | $SED \
    -e "s/txtUsername=/txtUsername=$USERNAME/" \
    -e "s/Password=/Password=$PASSWORD/"`
  # extract the action url.  Adjustments can be to the url in the pipeline
  ACTION=`echo "$FORM" | $TAIL -1`

  # Break up the effective url based on the /'s
  OFS="$IFS"
  IFS="/"
  set $effect
  IFS="$OFS"

  # Assemble the proper action url.  It should be one of these forms:
  # Complete URL:  http://system.domain.com/path/to/form/processor
  # URL="$ACTION"
  # Absolute path, no hostname: /path/to/form/processor
  # URL="$1//$3$ACTION"
  # Relative path, no hostname: path/to/form/processor
  # URL="$1//$3/$ACTION"
  # pick up $1 (protocol) & $3 (hostname), because the // in the effective
url makes $2=null.
  URL="$1//$3$ACTION"

  STEP=step2
  FILENAME=$BBDIR/$TESTHOST.$TEST.$STEP.html
  rm -f $FILENAME

  # Post the form back to the action url.  The returned file should
  # contain the page a browser would show after logging in.
  effect=`$CURL $STDOPTS \
    -w "%{url_effective}" \
    -o $FILENAME \
    $FORMVARS \
    $URL`
  ret=$?

  # As before, make sure the file is not empty, and that it contains
  # something we recognise from a successful login.  You can also look
  # for signs of login failure, such as a repeat of the login page, or
  # some message such as "login failed".

  if [ -s $FILENAME ]; then
    if [ "$ret" -eq "0" ]; then
      TESTSTR="Big Brother Test Page"
      check=`$GREP -ic "$TESTSTR" $FILENAME`
      if [ "$check" -gt "0" ]; then
        MESSAGE="$MESSAGE
$STEP: Found <b>$TESTSTR</b>.  Looking good"
      fi
    else
      MESSAGE="$MESSAGE
$STEP: failed to find <b>$TESTSTR</b>.  curl error code: $ret.  Url: <a
href=$URL>$URL</a>"
      bailout=1
    fi
  else
    MESSAGE-"$MESSAGE
$STEP: failed with empty file.  curl error code: $ret.  Url: <a
href=$URL>$URL</a>"
    bailout=1
  fi
fi

# If bailout is non-zero, something unexpected happened.
if [ "$bailout" -eq "1" ]; then
  COLOR="red"
elif [ "$bailout" -eq "2" ]; then
  # This is useful for non-critical-but-interesting conditions
  # that may be worth looking for.  There are none in this example.
  COLOR="yellow"
else
  # WooHoo!!
  COLOR="green"
fi

MESSAGE="$MESSAGE
<P>
See the saved files in <a href=$BBWEBHOST/bb/siteminder/>the checkout
directory</a>
<P>
<a href=$ORIGURL>Manual Check</a>"
#

# The script runs from cron, with output directed to a file in /tmp.
# This echo is not strictly necessary, but it can be useful to validate
# that it ran correctly
echo "$TESTHOST: $TEST: $COLOR"

# convert the dots to commas in the hostname
MACHINE=`echo $TESTHOST | $SED -e 's/\./,/g'`

# Assemble the status message as per normal
LINE="status $MACHINE.$TEST $COLOR `date`
$MESSAGE"

# fire it off to BB/Hobbit
$BB $BBDISP "$LINE"                     # SEND IT TO BBDISPLAY

========== formextract.pl ==========

#!/usr/bin/env perl
# $Id: formfind,v 1.5 2003/04/28 13:48:16 bagder Exp $
#
# formfind.pl
#
# This script gets a HTML page on stdin and presents form information on
# stdout.
#
# Author: Daniel Stenberg <daniel (at) haxx.se>
# Version: 0.2 Nov 18, 2002
#
# HISTORY
#
# 0.1 - Nov 12 1998 - Created now!
# 0.2 - Nov 18 2002 - Enhanced. Removed URL support, use only stdin.
# sometime          - echo the first form in a format suitable for dropping
#                     into a curl command line.   - Ralph Mitchell
#

$in="";

if($ARGV[0] eq "-h") {
    print  "Usage: $0 < HTML\n";
    exit;
}

sub namevalue {
    my ($tag)= (at) _;
    my $name=$tag;
    if($name =~ /name *=/i) {
        if($name =~ /name *= *([^\"\']([^ \">]*))/i) {
            $name = $1;
        }
        elsif($name =~ /name *= *(\"|\')([^\"\']*)(\"|\')/i) {
            $name=$2;
        }
        else {
            # there is a tag but we didn't find the contents
            $name="[weird]";
        }

    }
    else {
        # no name given
        $name="";
    }
    # get value tag
    my $value= $tag;
    if($value =~ /[^\.a-zA-Z0-9]value *=/i) {
        if($value =~ /[^\.a-zA-Z0-9]value *= *([^\"\']([^ \">]*))/i) {
            $value = $1;
        }
        elsif($value =~ /[^\.a-zA-Z0-9]value *= *(\"|\')([^\"\']*)(\"|\')/i)
{
            $value=$2;
        }
        else {
            # there is a tag but we didn't find the contents
            $value="[weird]";
        }
    }
    else {
        $value="";
    }
    return ($name, $value);
}


while(<STDIN>) {
    $line = $_;
    push @indoc, $line;
    $line=~ s/\n//g;
    $line=~ s/\r//g;
    $in=$in.$line;
}

while($in =~ /[^<]*(<[^>]+>)/g ) {
    # we have a tag in $1
    $tag = $1;
    if($tag =~ /^<!--/) {
        # this is a comment tag, ignore it
    }
    else {
        if(!$form &&
           ($tag =~ /^< *form/i )) {
            $method= $tag;
            if($method =~ /method *=/i) {
                $method=~ s/.*method *= *(\"|)([^ \">]*).*/$2/gi;
            }
            else {
                $method="get"; # default method
            }
            $action= $tag;
            $action=~ s/.*action *= *(\'|\"|)([^ \"\'>]*).*/$2/gi;

            $method=uc($method);

            $enctype=$tag;
            if ($enctype =~ /enctype *=/) {
                $enctype=~ s/.*enctype *= *(\'|\"|)([^ \"\'>]*).*/$2/gi;

                if($enctype eq "multipart/form-data") {
                    $enctype="multipart form upload [use -F]"
    }
                $enctype = "\n--- type: $enctype";
            }
            else {
                $enctype="";
            }

            # print "URL=\"$action\"$enctype\n";
            $form=1;
        }
        elsif($form &&
              ($tag =~ /< *\/form/i )) {

            # print "--- end of FORM\n";
            print "$action\n";
            $form=0;
            if( 0 ) {
                print "*** Fill in all or any of these: (default assigns may
be shown)\n";
                for(@vars) {
                    $var = $_;
                    $def = $value{$var};
                    print "$var=$def\n";
                }
                print "*** Pick one of these:\n";
                for(@alts) {
                    print "$_\n";
                }
            }
            undef @vars;
            undef @alts;
        }
        elsif($form &&
              ($tag =~ /^< *(input|select)/i)) {
            $mtag = $1;

            ($name, $value)=namevalue($tag);
            if($mtag =~ /select/i) {
                print "Select: NAME=\"$name\"\n";
                push @vars, "$name";
                $select = 1;
            }
            else {
                $type=$tag;
                if($type =~ /type *=/i) {
                    $type =~ s/.*type *= *(\'|\"|)([^ \"\'>]*).*/$2/gi;
                }
                else {
                    $type="text"; # default type
                }
                $type=uc($type);
                if(lc($type) eq "reset") {
                    # reset types are for UI only, ignore.
                }
                elsif($name eq "") {
                    # let's read the value parameter

                    print "Button: \"$value\" ($type)\n";
                    push @alts, "$value";
                }
                else {
                    $xname = s/\ /\+/g;
                    print "  -d $name=";
                    if($value ne "") {
                        # print "\"$value\"";
                        print "$value";
                    }
                    print " \n";
                    push @vars, "$name";
                    # store default value:
                    $value{$name}=$value;
                }
            }
        }
        elsif($form &&
              ($tag =~ /^< *textarea/i)) {
            my ($name, $value)=namevalue($tag);

            print "Textarea: NAME=\"$name\"\n";
        }
        elsif($select) {
            if($tag =~ /^< *\/ *select/i) {
                print "[end of select]\n";
                $select = 0;
            }
            elsif($tag =~ /[^\/] *option/i ) {
                my ($name, $value)=namevalue($tag);
                my $s;
                if($tag =~ /selected/i) {
                    $s= " (SELECTED)";
                }
                print "  Option VALUE=\"$value\"$s\n";
            }
        }
    }
}

====================