can a program be used to fill out a form on a website?

I need to populate a form on another website from a program or script.

[Meryl_Smith]

I've been asked to create a script or executable that can populate a form on another website using data accessible to the program.

For example, how would you create a program or script to fill in the advanced search form on yahoo, given that the script used a database to get the field contents to fill in?

Is there a way that you could capture the results of the page, as well?

Thanks!
Meryl

[dmorison]

Hi,

Study the HTTP protocol, and learn how GET and POST work. Nothing much to it, it's a simple character based protocol.

Then you simply have to look at the forms you wish to automate and have your program fire up a socket connection to the web server on port 80 and send the appropriate GET or POST commands that would submit the form.

The web server will send you a reply back, which you can just read in as a stream from your socket.

This is a simplistic view and would work for very basic forms. If you're using a service that has required a login then you may need to maintain a session with the web server and handle cookies. This can get tricky, but there a few "web scripting" commercial packages. A Google search should find something!

[toadhall]

What is the opinion of WWer's on this (especially "capture the results of the page")?

It seems the ethical/legal grey veil extends well over it. Some may see it as innocuously automating a normal process. Others may see it as criminal ("stealing content").

Would you caution Meryl before proceeding? That is, "prepare to be banned"?

T

[ziggystardust]

If I was going to use the function (available for anyone), on my site, I'd definetly ask for permission first or at least tell my users where the results come from and link to the other site.

That is if I was putting it up on a public website. However, If it's an envoirment where you're in control of the number of requests (like an administration system with only ten authorized users), I'd just go ahead and do it. It's not really an ethical issue when we're talking 50 requests a day (that is my opinion ;)

Either way, good luck.
//ZS

[carfac]

>>>> It seems the ethical/legal grey veil extends well over it

This is why a lot of sites are going to a user-input-from-an-image sort of test before the form will be accepted. (See an add a url page at alta vista, as an example)

The images use weirdly-formed letters.... something us smart humans can decipher, but something hard for a computer OCR to decode.

This makes it hard (if not impossible) for computer submissions.

dave

[toadhall]

It would seem the content to be captured is important enough to access often enough to warrant automation so I would hope Meryl and Co. don't end up shooting themselves in the foot by being banned from accessing the material with even a *manual* query.

What do all you seasoned and well placed Webmasters with the "almost perfect .htaccess" IP banning hammer have to say to her?

An ounce of prevention and all that.

T

[Meryl_Smith]

Thanks for the discussion on my question!

The interface we are trying to develop is for a particular business reporting requirement. We will be filling in information which our business customers would have had to type in manually, each day, on a website, to report and track activity which is stored in their operations system database (which we developed) on their business computers.

It may turn out that the website we need to report to expects an xml-formatted message, so this may now be a moot point.... I am currently gathering more information from the company that owns the website.

Thanks, again! Depending on their requirements, I may still need help.

Meryl