<div class="gmail_quote">On Mon, May 25, 2009 at 10:23 AM, Ralph Mitchell <span dir="ltr"><<a href="mailto:ralphmitchell@gmail.com">ralphmitchell@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
It was titled "Hobbit content check script". Here's the body of the message, and the attachments:<div><br></div><div><br></div><div><span style="border-collapse:collapse;font-size:16px"><div style="font-size:80%;margin-top:5px;margin-right:15px;margin-bottom:5px;margin-left:15px;padding-bottom:20px">
Sorry I didn't get back to you earlier - I've been fighting with perl<br>& SOAP, trying to get some status messages out of a bunch of load<br>balancers. I think I finally got it now....<br><br>Attached (hopefully :) are a couple of files I tried to post to the<br>
list some time last year. There's a shell script that does a login to<br>a <span style="background-repeat:initial;background-color:rgb(255, 255, 136)">Siteminder</span> site, and a perl script evolved from something Daniel<br>
Stenberg wrote - he's the original author of curl. Daniel's very<br>active on the curl mailing list, by the way, along with a bunch of<br>other helpful folks, if it starts going weird on you.<br><br>The <span style="background-repeat:initial;background-color:rgb(255, 255, 136)">Siteminder</span> login probably won't be much use to you as is, but it<br>
demonstrates a few things I've worked out over the years. Like, the<br>standard options string to pass to curl, collecting messages along the<br>way, and loading up the message string with a Manual Check url at the<br>
end for ops to try the check for themselves before waking people up...<br><br>You can put almost *anything* into the message string that gets sent<br>to Hobbit. Right now I have scripts that check car rentals at travel<br>
sites for a particular company and emit a table of the current prices.<br>Another script hits a cruise monitor and puts out a list using the<br>&red and &green dots to mark cruise lines as reachable or not. You<br>
can push out font color and size changes, urls to saved pages, etc.<br><br>As far as cookies go, I generally just have curl save them whenever it finishes:<br><br> curl -b cookies -c cookies .....<br><br>If you do that you can go look at the cookies and see what came in.<br>
Other things become possible too, such as editing cookies in between<br>curl calls. If you do keep cookies, though, the *first* thing you<br>should do in the script is delete the cookie file. That's like<br>clearing the browser cookie cache, and can avoid a lot of mysterious<br>
errors.<br><br>Finally, if you think you might spend some time on web site scripts,<br>there's a Firefox extension that is absolutely invaluable -<br>LiveHTTPheaders. It shows you *exactly* what the browser sends to the<br>
server, as well as what it gets back - not the page content, just the<br>headers, but that's all you need to recreate the sequence. It even<br>works twith secure sites. If you see something going out that you<br>can't account for, the usual suspect is javascript. I've seen<br>
javascript that assembles a url on the fly from form elements, fiddles<br>with some of the form elements, then posts the form. That kind of<br>thing can be difficult to nail down, and almost impossible to<br>replicate...<br>
<font color="#888888"><br>Ralph Mitchell</font></div></span></div></blockquote><div><br></div><div>The attachments may not have gone through, so here they are again:</div><div><br></div><div>========== bb-siteminder.sh ==========</div>
<div><div>#!/bin/sh</div><div><br></div><div># In case of disaster, uncomment the next line and watch the output</div><div># set -x</div><div>#</div><div># This script runs out of cron. Tell it where to find Big Brother.</div>
<div>BBHOME=/home/bb/bb18b3</div><div>export BBHOME</div><div><br></div><div>TESTHOST=<a href="http://system.domain.com">system.domain.com</a></div><div>URL="http://$TESTHOST/"</div><div>ORIGURL="$URL"</div>
<div>USERNAME=bbusername</div><div>PASSWORD=bbsecretword</div><div><br></div><div># Where to stash the downloaded html files. These get patched into the BB</div><div># report so that the monitoring guys can click through and see what we got.</div>
<div>BBDIR="$BBHOME/www/siteminder"</div><div><br></div><div># Gonna need a cookie jar. Start off empty, so each checkout starts clean.</div><div>COOKIES="$BBDIR/$TESTHOST.cookies"</div><div>rm -f $COOKIES</div>
<div><br></div><div># Some standard curl flags. See the curl man page for details</div><div>STDOPTS="-w %{url_effective} -L --connect-timeout 60 --max-time 120 -s -S -b $COOKIES -c $COOKIES -A Mozilla/4.79"</div>
<div><br></div><div># This is just a tag for identifying environment dumps.</div><div>BBPROG=bb-siteminder.sh; export BBPROG</div><div><br></div><div># The next block came from the BB ext example script. If BBHOME is not set</div>
<div># there's no point continuing. If BBTMP is not set, load up the standard</div><div># environment from bbdef.sh. </div><div>if test "$BBHOME" = ""</div><div>then</div><div> echo "BBHOME is not set... exiting"</div>
<div> exit 1</div><div>fi</div><div><br></div><div>if test ! "$BBTMP" # GET DEFINITIONS IF NEEDED</div><div>then</div><div> # echo "*** LOADING BBDEF ***"</div><div>
. $BBHOME/etc/bbdef.sh # INCLUDE STANDARD DEFINITIONS</div><div>fi</div><div><br></div><div># This is used to label the result column</div><div>TEST="login"</div><div><br></div><div># Make sure bail out code is set.</div>
<div>bailout=0</div><div><br></div><div># ========== step 1 ==========</div><div># Grab the first page</div><div>STEP=step1</div><div># Name the file for the host, test and step, to make it unique</div><div>FILENAME=$BBDIR/$TESTHOST.$TEST.$STEP.html</div>
<div># remove any previous copy to avoid misunderstandings</div><div>rm -f $FILENAME</div><div><br></div><div># Grab the first page. The effective url is recorded for patching</div><div># into the action url is necessary</div>
<div>effect=`$CURL $STDOPTS \</div><div> -o $FILENAME \</div><div> $URL`</div><div># Save the return code, it might be important</div><div>ret=$?</div><div><br></div><div># code 52 means "the server sent nothing"</div>
<div>if [ "$ret" -eq "52" ]; then</div><div> # empty reply, try it again</div><div> MESSAGE="$STEP: failed with curl error code: 52 - Empty reply from server. Url: <a href=$URL>$URL</a>"</div>
<div> MESSAGE="$MESSAGE</div><div>$STEP: retrying..."</div><div> sleep 2</div><div> effect=`$CURL $STDOPTS \</div><div> -o /dev/null \</div><div> $URL`</div><div> ret=$?</div><div>fi</div><div><br></div>
<div>if [ -s $FILENAME ]; then</div><div> # file exists and has size greater than zero. look for something to validate</div><div> TESTSTR="Something from the Login Page"</div><div> check=`$GREP -ic "$TESTSTR" $FILENAME`</div>
<div> if [ "$check" -gt "0" ]; then</div><div> # Found the test string, we're good to continue</div><div> MESSAGE="$MESSAGE</div><div>$STEP: Got Login page - found <b>$TESTSTR</b>"</div>
<div> else</div><div> # Didn't find the test string, bail out of subsequent steps</div><div> MESSAGE="$MESSAGE</div><div>$STEP: failed to find <b>$TESTSTR</b>. curl error code: $ret. Url: <a href=$URL>$URL</a>"</div>
<div> bailout=1</div><div> fi</div><div>else</div><div> # Got nothing useful from the server, give up</div><div> MESSAGE-"$MESSAGE</div><div>$STEP: failed with empty file. curl error code: $ret. Url: <a href=$URL>$URL</a>"</div>
<div> bailout=1</div><div>fi</div><div><br></div><div># If previous step completed OK, perform next step</div><div>if [ "$bailout" -eq "0" ]; then</div><div> # formextract.pl strips out the form elements and formats them like this:</div>
<div> # -d username=</div><div> # -d password=</div><div> # <a href="http://server.domain.com/path/to/form/processor">http://server.domain.com/path/to/form/processor</a></div><div> # On some pages, the action url is not fully qualified.</div>
<div> FORM=`cat $FILENAME | $HOME/bin/formextract.pl`</div><div> # Pull out the form variable lines and patch in the userid & password</div><div> FORMVARS=`echo "$FORM" | $GREP -- '-d' | $SED \</div>
<div> -e "s/txtUsername=/txtUsername=$USERNAME/" \</div><div> -e "s/Password=/Password=$PASSWORD/"`</div><div> # extract the action url. Adjustments can be to the url in the pipeline</div><div>
ACTION=`echo "$FORM" | $TAIL -1`</div><div><br></div><div> # Break up the effective url based on the /'s</div><div> OFS="$IFS"</div><div> IFS="/"</div><div> set $effect</div><div> IFS="$OFS"</div>
<div><br></div><div> # Assemble the proper action url. It should be one of these forms:</div><div> # Complete URL: <a href="http://system.domain.com/path/to/form/processor">http://system.domain.com/path/to/form/processor</a></div>
<div> # URL="$ACTION"</div><div> # Absolute path, no hostname: /path/to/form/processor</div><div> # URL="$1//$3$ACTION"</div><div> # Relative path, no hostname: path/to/form/processor</div><div> # URL="$1//$3/$ACTION"</div>
<div> # pick up $1 (protocol) & $3 (hostname), because the // in the effective url makes $2=null.</div><div> URL="$1//$3$ACTION"</div><div><br></div><div> STEP=step2</div><div> FILENAME=$BBDIR/$TESTHOST.$TEST.$STEP.html</div>
<div> rm -f $FILENAME</div><div><br></div><div> # Post the form back to the action url. The returned file should</div><div> # contain the page a browser would show after logging in.</div><div> effect=`$CURL $STDOPTS \</div>
<div> -w "%{url_effective}" \</div><div> -o $FILENAME \</div><div> $FORMVARS \</div><div> $URL`</div><div> ret=$?</div><div><br></div><div> # As before, make sure the file is not empty, and that it contains</div>
<div> # something we recognise from a successful login. You can also look</div><div> # for signs of login failure, such as a repeat of the login page, or</div><div> # some message such as "login failed".</div>
<div><br></div><div> if [ -s $FILENAME ]; then</div><div> if [ "$ret" -eq "0" ]; then</div><div> TESTSTR="Big Brother Test Page"</div><div> check=`$GREP -ic "$TESTSTR" $FILENAME`</div>
<div> if [ "$check" -gt "0" ]; then</div><div> MESSAGE="$MESSAGE</div><div>$STEP: Found <b>$TESTSTR</b>. Looking good"</div><div> fi</div><div> else</div><div>
MESSAGE="$MESSAGE</div><div>$STEP: failed to find <b>$TESTSTR</b>. curl error code: $ret. Url: <a href=$URL>$URL</a>"</div><div> bailout=1</div><div> fi</div><div> else</div>
<div> MESSAGE-"$MESSAGE</div><div>$STEP: failed with empty file. curl error code: $ret. Url: <a href=$URL>$URL</a>"</div><div> bailout=1</div><div> fi</div><div>fi</div><div><br></div><div># If bailout is non-zero, something unexpected happened.</div>
<div>if [ "$bailout" -eq "1" ]; then</div><div> COLOR="red"</div><div>elif [ "$bailout" -eq "2" ]; then</div><div> # This is useful for non-critical-but-interesting conditions</div>
<div> # that may be worth looking for. There are none in this example.</div><div> COLOR="yellow"</div><div>else</div><div> # WooHoo!!</div><div> COLOR="green"</div><div>fi</div><div><br></div><div>
MESSAGE="$MESSAGE</div><div><P></div><div>See the saved files in <a href=$BBWEBHOST/bb/siteminder/>the checkout directory</a></div><div><P></div><div><a href=$ORIGURL>Manual Check</a>"</div>
<div>#</div><div><br></div><div># The script runs from cron, with output directed to a file in /tmp.</div><div># This echo is not strictly necessary, but it can be useful to validate</div><div># that it ran correctly</div>
<div>echo "$TESTHOST: $TEST: $COLOR"</div><div><br></div><div># convert the dots to commas in the hostname</div><div>MACHINE=`echo $TESTHOST | $SED -e 's/\./,/g'`</div><div><br></div><div># Assemble the status message as per normal</div>
<div>LINE="status $MACHINE.$TEST $COLOR `date`</div><div>$MESSAGE"</div><div><br></div><div># fire it off to BB/Hobbit</div><div>$BB $BBDISP "$LINE" # SEND IT TO BBDISPLAY</div><div>
<br></div><div>========== formextract.pl ==========</div><div><br></div><div><div>#!/usr/bin/env perl</div><div># $Id: formfind,v 1.5 2003/04/28 13:48:16 bagder Exp $</div><div>#</div><div># formfind.pl</div><div>#</div><div>
# This script gets a HTML page on stdin and presents form information on</div><div># stdout.</div><div>#</div><div># Author: Daniel Stenberg <<a href="mailto:daniel@haxx.se">daniel@haxx.se</a>></div><div># Version: 0.2 Nov 18, 2002</div>
<div>#</div><div># HISTORY</div><div>#</div><div># 0.1 - Nov 12 1998 - Created now!</div><div># 0.2 - Nov 18 2002 - Enhanced. Removed URL support, use only stdin.</div><div># sometime - echo the first form in a format suitable for dropping</div>
<div># into a curl command line. - Ralph Mitchell</div><div>#</div><div><br></div><div>$in="";</div><div><br></div><div>if($ARGV[0] eq "-h") {</div><div> print "Usage: $0 < HTML\n";</div>
<div> exit;</div><div>}</div><div><br></div><div>sub namevalue {</div><div> my ($tag)=@_;</div><div> my $name=$tag;</div><div> if($name =~ /name *=/i) {</div><div> if($name =~ /name *= *([^\"\']([^ \">]*))/i) {</div>
<div> $name = $1;</div><div> }</div><div> elsif($name =~ /name *= *(\"|\')([^\"\']*)(\"|\')/i) {</div><div> $name=$2;</div><div> }</div><div> else {</div>
<div> # there is a tag but we didn't find the contents</div><div> $name="[weird]";</div><div> }</div><div><br></div><div> }</div><div> else {</div><div> # no name given</div>
<div> $name="";</div><div> }</div><div> # get value tag</div><div> my $value= $tag;</div><div> if($value =~ /[^\.a-zA-Z0-9]value *=/i) {</div><div> if($value =~ /[^\.a-zA-Z0-9]value *= *([^\"\']([^ \">]*))/i) {</div>
<div> $value = $1;</div><div> }</div><div> elsif($value =~ /[^\.a-zA-Z0-9]value *= *(\"|\')([^\"\']*)(\"|\')/i) {</div><div> $value=$2;</div><div> }</div>
<div> else {</div><div> # there is a tag but we didn't find the contents</div><div> $value="[weird]";</div><div> }</div><div> }</div><div> else {</div><div> $value="";</div>
<div> }</div><div> return ($name, $value);</div><div>}</div><div><br></div><div><br></div><div>while(<STDIN>) {</div><div> $line = $_;</div><div> push @indoc, $line;</div><div> $line=~ s/\n//g;</div><div>
$line=~ s/\r//g;</div><div> $in=$in.$line;</div><div>}</div><div><br></div><div>while($in =~ /[^<]*(<[^>]+>)/g ) {</div><div> # we have a tag in $1</div><div> $tag = $1;</div><div><span class="Apple-tab-span" style="white-space:pre"> </span></div>
<div> if($tag =~ /^<!--/) {</div><div> # this is a comment tag, ignore it </div><div> }</div><div> else {</div><div> if(!$form &&</div><div> ($tag =~ /^< *form/i )) {</div><div>
$method= $tag;</div><div> if($method =~ /method *=/i) {</div><div> $method=~ s/.*method *= *(\"|)([^ \">]*).*/$2/gi;</div><div> }</div><div> else {</div>
<div> $method="get"; # default method</div><div> }</div><div> $action= $tag;</div><div> $action=~ s/.*action *= *(\'|\"|)([^ \"\'>]*).*/$2/gi;</div>
<div> </div><div> $method=uc($method);</div><div> </div><div> $enctype=$tag;</div><div> if ($enctype =~ /enctype *=/) {</div><div> $enctype=~ s/.*enctype *= *(\'|\"|)([^ \"\'>]*).*/$2/gi;</div>
<div> </div><div> if($enctype eq "multipart/form-data") {</div><div> $enctype="multipart form upload [use -F]"</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> }</div>
<div> $enctype = "\n--- type: $enctype";</div><div> }</div><div> else {</div><div> $enctype="";</div><div> }</div><div><br></div><div> # print "URL=\"$action\"$enctype\n";</div>
<div> $form=1;</div><div> }</div><div> elsif($form &&</div><div> ($tag =~ /< *\/form/i )) {</div><div><br></div><div> # print "--- end of FORM\n";</div>
<div> print "$action\n";</div><div> $form=0;</div><div> if( 0 ) {</div><div> print "*** Fill in all or any of these: (default assigns may be shown)\n";</div>
<div> for(@vars) {</div><div> $var = $_;</div><div> $def = $value{$var};</div><div> print "$var=$def\n";</div><div> }</div><div>
print "*** Pick one of these:\n";</div><div> for(@alts) {</div><div> print "$_\n";</div><div> }</div><div> }</div><div> undef @vars;</div>
<div> undef @alts;</div><div> }</div><div> elsif($form &&</div><div> ($tag =~ /^< *(input|select)/i)) {</div><div> $mtag = $1;</div><div> </div><div> ($name, $value)=namevalue($tag);</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span></div><div> if($mtag =~ /select/i) {</div><div> print "Select: NAME=\"$name\"\n";</div><div> push @vars, "$name";</div>
<div> $select = 1;</div><div> }</div><div> else {</div><div> $type=$tag;</div><div> if($type =~ /type *=/i) {</div><div> $type =~ s/.*type *= *(\'|\"|)([^ \"\'>]*).*/$2/gi;</div>
<div> }</div><div> else {</div><div> $type="text"; # default type</div><div> }</div><div> $type=uc($type);</div><div> if(lc($type) eq "reset") {</div>
<div> # reset types are for UI only, ignore.</div><div> }</div><div> elsif($name eq "") {</div><div> # let's read the value parameter</div>
<div> </div><div> print "Button: \"$value\" ($type)\n";</div><div> push @alts, "$value";</div><div> }</div><div> else {</div>
<div> $xname = s/\ /\+/g;</div><div> print " -d $name=";</div><div> if($value ne "") {</div><div> # print "\"$value\"";</div>
<div> print "$value";</div><div> }</div><div> print " \n";</div><div> push @vars, "$name";</div><div> # store default value:</div>
<div> $value{$name}=$value;</div><div> }</div><div> }</div><div> }</div><div> elsif($form &&</div><div> ($tag =~ /^< *textarea/i)) {</div>
<div> my ($name, $value)=namevalue($tag);</div><div> </div><div> print "Textarea: NAME=\"$name\"\n";</div><div> }</div><div> elsif($select) {</div><div> if($tag =~ /^< *\/ *select/i) {</div>
<div> print "[end of select]\n";</div><div> $select = 0;</div><div> }</div><div> elsif($tag =~ /[^\/] *option/i ) {</div><div> my ($name, $value)=namevalue($tag);</div>
<div> my $s;</div><div> if($tag =~ /selected/i) {</div><div> $s= " (SELECTED)";</div><div> }</div><div> print " Option VALUE=\"$value\"$s\n";</div>
<div> }</div><div> }</div><div> }</div><div>}</div><div><br></div><div>====================</div></div></div></div>