[hobbit] Http login test
Ralph Mitchell
ralphmitchell at gmail.com
Mon May 25 17:51:57 CEST 2009
On Mon, May 25, 2009 at 10:23 AM, Ralph Mitchell <ralphmitchell at gmail.com>wrote:
> It was titled "Hobbit content check script". Here's the body of the
> message, and the attachments:
>
> Sorry I didn't get back to you earlier - I've been fighting with perl
> & SOAP, trying to get some status messages out of a bunch of load
> balancers. I think I finally got it now....
>
> Attached (hopefully :) are a couple of files I tried to post to the
> list some time last year. There's a shell script that does a login to
> a Siteminder site, and a perl script evolved from something Daniel
> Stenberg wrote - he's the original author of curl. Daniel's very
> active on the curl mailing list, by the way, along with a bunch of
> other helpful folks, if it starts going weird on you.
>
> The Siteminder login probably won't be much use to you as is, but it
> demonstrates a few things I've worked out over the years. Like, the
> standard options string to pass to curl, collecting messages along the
> way, and loading up the message string with a Manual Check url at the
> end for ops to try the check for themselves before waking people up...
>
> You can put almost *anything* into the message string that gets sent
> to Hobbit. Right now I have scripts that check car rentals at travel
> sites for a particular company and emit a table of the current prices.
> Another script hits a cruise monitor and puts out a list using the
> &red and &green dots to mark cruise lines as reachable or not. You
> can push out font color and size changes, urls to saved pages, etc.
>
> As far as cookies go, I generally just have curl save them whenever it
> finishes:
>
> curl -b cookies -c cookies .....
>
> If you do that you can go look at the cookies and see what came in.
> Other things become possible too, such as editing cookies in between
> curl calls. If you do keep cookies, though, the *first* thing you
> should do in the script is delete the cookie file. That's like
> clearing the browser cookie cache, and can avoid a lot of mysterious
> errors.
>
> Finally, if you think you might spend some time on web site scripts,
> there's a Firefox extension that is absolutely invaluable -
> LiveHTTPheaders. It shows you *exactly* what the browser sends to the
> server, as well as what it gets back - not the page content, just the
> headers, but that's all you need to recreate the sequence. It even
> works twith secure sites. If you see something going out that you
> can't account for, the usual suspect is javascript. I've seen
> javascript that assembles a url on the fly from form elements, fiddles
> with some of the form elements, then posts the form. That kind of
> thing can be difficult to nail down, and almost impossible to
> replicate...
>
> Ralph Mitchell
>
The attachments may not have gone through, so here they are again:
========== bb-siteminder.sh ==========
#!/bin/sh
# In case of disaster, uncomment the next line and watch the output
# set -x
#
# This script runs out of cron. Tell it where to find Big Brother.
BBHOME=/home/bb/bb18b3
export BBHOME
TESTHOST=system.domain.com
URL="http://$TESTHOST/"
ORIGURL="$URL"
USERNAME=bbusername
PASSWORD=bbsecretword
# Where to stash the downloaded html files. These get patched into the BB
# report so that the monitoring guys can click through and see what we got.
BBDIR="$BBHOME/www/siteminder"
# Gonna need a cookie jar. Start off empty, so each checkout starts clean.
COOKIES="$BBDIR/$TESTHOST.cookies"
rm -f $COOKIES
# Some standard curl flags. See the curl man page for details
STDOPTS="-w %{url_effective} -L --connect-timeout 60 --max-time 120 -s -S -b
$COOKIES -c $COOKIES -A Mozilla/4.79"
# This is just a tag for identifying environment dumps.
BBPROG=bb-siteminder.sh; export BBPROG
# The next block came from the BB ext example script. If BBHOME is not set
# there's no point continuing. If BBTMP is not set, load up the standard
# environment from bbdef.sh.
if test "$BBHOME" = ""
then
echo "BBHOME is not set... exiting"
exit 1
fi
if test ! "$BBTMP" # GET DEFINITIONS IF NEEDED
then
# echo "*** LOADING BBDEF ***"
. $BBHOME/etc/bbdef.sh # INCLUDE STANDARD DEFINITIONS
fi
# This is used to label the result column
TEST="login"
# Make sure bail out code is set.
bailout=0
# ========== step 1 ==========
# Grab the first page
STEP=step1
# Name the file for the host, test and step, to make it unique
FILENAME=$BBDIR/$TESTHOST.$TEST.$STEP.html
# remove any previous copy to avoid misunderstandings
rm -f $FILENAME
# Grab the first page. The effective url is recorded for patching
# into the action url is necessary
effect=`$CURL $STDOPTS \
-o $FILENAME \
$URL`
# Save the return code, it might be important
ret=$?
# code 52 means "the server sent nothing"
if [ "$ret" -eq "52" ]; then
# empty reply, try it again
MESSAGE="$STEP: failed with curl error code: 52 - Empty reply from server.
Url: <a href=$URL>$URL</a>"
MESSAGE="$MESSAGE
$STEP: retrying..."
sleep 2
effect=`$CURL $STDOPTS \
-o /dev/null \
$URL`
ret=$?
fi
if [ -s $FILENAME ]; then
# file exists and has size greater than zero. look for something to
validate
TESTSTR="Something from the Login Page"
check=`$GREP -ic "$TESTSTR" $FILENAME`
if [ "$check" -gt "0" ]; then
# Found the test string, we're good to continue
MESSAGE="$MESSAGE
$STEP: Got Login page - found <b>$TESTSTR</b>"
else
# Didn't find the test string, bail out of subsequent steps
MESSAGE="$MESSAGE
$STEP: failed to find <b>$TESTSTR</b>. curl error code: $ret. Url: <a
href=$URL>$URL</a>"
bailout=1
fi
else
# Got nothing useful from the server, give up
MESSAGE-"$MESSAGE
$STEP: failed with empty file. curl error code: $ret. Url: <a
href=$URL>$URL</a>"
bailout=1
fi
# If previous step completed OK, perform next step
if [ "$bailout" -eq "0" ]; then
# formextract.pl strips out the form elements and formats them like this:
# -d username=
# -d password=
# http://server.domain.com/path/to/form/processor
# On some pages, the action url is not fully qualified.
FORM=`cat $FILENAME | $HOME/bin/formextract.pl`
# Pull out the form variable lines and patch in the userid & password
FORMVARS=`echo "$FORM" | $GREP -- '-d' | $SED \
-e "s/txtUsername=/txtUsername=$USERNAME/" \
-e "s/Password=/Password=$PASSWORD/"`
# extract the action url. Adjustments can be to the url in the pipeline
ACTION=`echo "$FORM" | $TAIL -1`
# Break up the effective url based on the /'s
OFS="$IFS"
IFS="/"
set $effect
IFS="$OFS"
# Assemble the proper action url. It should be one of these forms:
# Complete URL: http://system.domain.com/path/to/form/processor
# URL="$ACTION"
# Absolute path, no hostname: /path/to/form/processor
# URL="$1//$3$ACTION"
# Relative path, no hostname: path/to/form/processor
# URL="$1//$3/$ACTION"
# pick up $1 (protocol) & $3 (hostname), because the // in the effective
url makes $2=null.
URL="$1//$3$ACTION"
STEP=step2
FILENAME=$BBDIR/$TESTHOST.$TEST.$STEP.html
rm -f $FILENAME
# Post the form back to the action url. The returned file should
# contain the page a browser would show after logging in.
effect=`$CURL $STDOPTS \
-w "%{url_effective}" \
-o $FILENAME \
$FORMVARS \
$URL`
ret=$?
# As before, make sure the file is not empty, and that it contains
# something we recognise from a successful login. You can also look
# for signs of login failure, such as a repeat of the login page, or
# some message such as "login failed".
if [ -s $FILENAME ]; then
if [ "$ret" -eq "0" ]; then
TESTSTR="Big Brother Test Page"
check=`$GREP -ic "$TESTSTR" $FILENAME`
if [ "$check" -gt "0" ]; then
MESSAGE="$MESSAGE
$STEP: Found <b>$TESTSTR</b>. Looking good"
fi
else
MESSAGE="$MESSAGE
$STEP: failed to find <b>$TESTSTR</b>. curl error code: $ret. Url: <a
href=$URL>$URL</a>"
bailout=1
fi
else
MESSAGE-"$MESSAGE
$STEP: failed with empty file. curl error code: $ret. Url: <a
href=$URL>$URL</a>"
bailout=1
fi
fi
# If bailout is non-zero, something unexpected happened.
if [ "$bailout" -eq "1" ]; then
COLOR="red"
elif [ "$bailout" -eq "2" ]; then
# This is useful for non-critical-but-interesting conditions
# that may be worth looking for. There are none in this example.
COLOR="yellow"
else
# WooHoo!!
COLOR="green"
fi
MESSAGE="$MESSAGE
<P>
See the saved files in <a href=$BBWEBHOST/bb/siteminder/>the checkout
directory</a>
<P>
<a href=$ORIGURL>Manual Check</a>"
#
# The script runs from cron, with output directed to a file in /tmp.
# This echo is not strictly necessary, but it can be useful to validate
# that it ran correctly
echo "$TESTHOST: $TEST: $COLOR"
# convert the dots to commas in the hostname
MACHINE=`echo $TESTHOST | $SED -e 's/\./,/g'`
# Assemble the status message as per normal
LINE="status $MACHINE.$TEST $COLOR `date`
$MESSAGE"
# fire it off to BB/Hobbit
$BB $BBDISP "$LINE" # SEND IT TO BBDISPLAY
========== formextract.pl ==========
#!/usr/bin/env perl
# $Id: formfind,v 1.5 2003/04/28 13:48:16 bagder Exp $
#
# formfind.pl
#
# This script gets a HTML page on stdin and presents form information on
# stdout.
#
# Author: Daniel Stenberg <daniel at haxx.se>
# Version: 0.2 Nov 18, 2002
#
# HISTORY
#
# 0.1 - Nov 12 1998 - Created now!
# 0.2 - Nov 18 2002 - Enhanced. Removed URL support, use only stdin.
# sometime - echo the first form in a format suitable for dropping
# into a curl command line. - Ralph Mitchell
#
$in="";
if($ARGV[0] eq "-h") {
print "Usage: $0 < HTML\n";
exit;
}
sub namevalue {
my ($tag)=@_;
my $name=$tag;
if($name =~ /name *=/i) {
if($name =~ /name *= *([^\"\']([^ \">]*))/i) {
$name = $1;
}
elsif($name =~ /name *= *(\"|\')([^\"\']*)(\"|\')/i) {
$name=$2;
}
else {
# there is a tag but we didn't find the contents
$name="[weird]";
}
}
else {
# no name given
$name="";
}
# get value tag
my $value= $tag;
if($value =~ /[^\.a-zA-Z0-9]value *=/i) {
if($value =~ /[^\.a-zA-Z0-9]value *= *([^\"\']([^ \">]*))/i) {
$value = $1;
}
elsif($value =~ /[^\.a-zA-Z0-9]value *= *(\"|\')([^\"\']*)(\"|\')/i)
{
$value=$2;
}
else {
# there is a tag but we didn't find the contents
$value="[weird]";
}
}
else {
$value="";
}
return ($name, $value);
}
while(<STDIN>) {
$line = $_;
push @indoc, $line;
$line=~ s/\n//g;
$line=~ s/\r//g;
$in=$in.$line;
}
while($in =~ /[^<]*(<[^>]+>)/g ) {
# we have a tag in $1
$tag = $1;
if($tag =~ /^<!--/) {
# this is a comment tag, ignore it
}
else {
if(!$form &&
($tag =~ /^< *form/i )) {
$method= $tag;
if($method =~ /method *=/i) {
$method=~ s/.*method *= *(\"|)([^ \">]*).*/$2/gi;
}
else {
$method="get"; # default method
}
$action= $tag;
$action=~ s/.*action *= *(\'|\"|)([^ \"\'>]*).*/$2/gi;
$method=uc($method);
$enctype=$tag;
if ($enctype =~ /enctype *=/) {
$enctype=~ s/.*enctype *= *(\'|\"|)([^ \"\'>]*).*/$2/gi;
if($enctype eq "multipart/form-data") {
$enctype="multipart form upload [use -F]"
}
$enctype = "\n--- type: $enctype";
}
else {
$enctype="";
}
# print "URL=\"$action\"$enctype\n";
$form=1;
}
elsif($form &&
($tag =~ /< *\/form/i )) {
# print "--- end of FORM\n";
print "$action\n";
$form=0;
if( 0 ) {
print "*** Fill in all or any of these: (default assigns may
be shown)\n";
for(@vars) {
$var = $_;
$def = $value{$var};
print "$var=$def\n";
}
print "*** Pick one of these:\n";
for(@alts) {
print "$_\n";
}
}
undef @vars;
undef @alts;
}
elsif($form &&
($tag =~ /^< *(input|select)/i)) {
$mtag = $1;
($name, $value)=namevalue($tag);
if($mtag =~ /select/i) {
print "Select: NAME=\"$name\"\n";
push @vars, "$name";
$select = 1;
}
else {
$type=$tag;
if($type =~ /type *=/i) {
$type =~ s/.*type *= *(\'|\"|)([^ \"\'>]*).*/$2/gi;
}
else {
$type="text"; # default type
}
$type=uc($type);
if(lc($type) eq "reset") {
# reset types are for UI only, ignore.
}
elsif($name eq "") {
# let's read the value parameter
print "Button: \"$value\" ($type)\n";
push @alts, "$value";
}
else {
$xname = s/\ /\+/g;
print " -d $name=";
if($value ne "") {
# print "\"$value\"";
print "$value";
}
print " \n";
push @vars, "$name";
# store default value:
$value{$name}=$value;
}
}
}
elsif($form &&
($tag =~ /^< *textarea/i)) {
my ($name, $value)=namevalue($tag);
print "Textarea: NAME=\"$name\"\n";
}
elsif($select) {
if($tag =~ /^< *\/ *select/i) {
print "[end of select]\n";
$select = 0;
}
elsif($tag =~ /[^\/] *option/i ) {
my ($name, $value)=namevalue($tag);
my $s;
if($tag =~ /selected/i) {
$s= " (SELECTED)";
}
print " Option VALUE=\"$value\"$s\n";
}
}
}
}
====================
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xymon.com/pipermail/xymon/attachments/20090525/973ee2ec/attachment.html>
More information about the Xymon
mailing list