Re: [hobbit] URLplus

["Ralph Mitchell" <ralphmitchell (at) gmail.com>]

On Wed, Apr 30, 2008 at 12:35 PM, Gary Baluha <gumby3203 (at) gmail.com> wrote:

> On Wed, Apr 30, 2008 at 1:28 PM, Ralph Mitchell <ralphmitchell (at) gmail.com>
> wrote:
>
> > Unfortunately it's not just one script.  I've found many variations in
> > login pages, so I've had to customize the scripts for each different
> > target.  Some logins require a form to be posted back, others are handled
> > via WWW-Authenticate pop-up boxes and sometimes redirects have to be
> > followed.  That's not hard if the site uses Location headers, because curl
> > can handle those, but I've also found meta-refresh tags and several types of
> > javascript "go-elsewhere" redirects that have to be individually scripted.
>
>
> And these are some of the same issues I've run in to while writing the
> URLPlus script.  I've been trying to slowly morph it into something that can
> configured solely through the configuration file without having to change
> any code, but that's easier said than done.  Especially since the original
> reason I was writing the script got removed, and is supposed to be converted
> to yet another monitoring system (which so far hasn't progressed, probably
> because of the fact that some of the monitors are non-trivial).
>

Seems like we have a "get-rid-of-Hobbit" project about every 2 years or so.


I think our National Tools Team has some kind of click-through tools that
records page requests, but  I'm pretty sure it doesn't do too well with
forms that include some variable(s) sent back by the server.  For example,
one thing I've seen is a form variable called __VIEWSTATE that seems to get
a new, longer value with each successive page loaded.  I've no idea what the
value is as it's encoded some way, but I suppose it tracks where you are in
the page.

I don't think the script recorder can handle variable inputs either.  I'm
watching a number of web pages where I have to enter a date that's days or
weeks ahead of today's date.  The recorder will just see the post containing
the future date and faithfully re-post it every time, right up until it
becomes a past date and the monitor breaks...

I won't be too surprised if in the end I get to keep my scripts, but I'll
probably have to send traps to Unicenter as well as sending Hobbit reports.
That way the "best shore" eyes-on-glass folks in Brazil or Singapore or
wherever won't have to deal with Hobbit, while the 2nd-level folks at home
still get the detail they're used to.

In the meantime, attached are the two files I sent previously, on 6/23/06.
Here's what I wrote then:


bb-siteminder.sh is an example script to demonstrate how to login
>
> formextract.pl is a slightly modified copy of Daniel Stenberg's
> formfind.pl, which can be found here:
>
>    http://curl.haxx.se/docs/programs.html
>
> I'm not a perl programmer, so any screwups in formextract are mine, not
> Daniel's... :)
>
> It's doubtful that the example script will work out of the box.  I've
> found that I generally have to take it in easy stages - fetch the first
> page, eyeball it, adjust the script, lather, rinse, repeat.
>


LiveHTTPHeaders is a really useful Firefox extension for tracking what
happens when navigating web pages.

Ralph Mitchell