Welcome to WebmasterWorld Guest from 54.145.39.186

Forum Moderators: coopster & jatar k

Message Too Old, No Replies

PHP Crawler Questions

Access content that is user & password protected

     

wfernley

10:22 pm on Oct 29, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Hey everyone,

I have a bit of an interesting project. I need to create a PHP web crawler that will automatically login to a site and grab content from web pages.

I have created crawlers before but I have never used them to access password protected pages. The pages are password protected via a HTML login form and encrypted via SSL.

Has anyone created something like this? The main stumbling block for me is just getting past the login form. I don't know how to create the script to automatically login to access the pages. Once logged in it would need to follow links.

The use of this script is completely legit. It is for a reseller looking to grab product information from their distributor. Considering there are thousands of products, they want to automate the process. Unfortunately, the distributor doesn't offer any type of feed.

Can anyone give me some advice on how to get it to work?

Thanks in advance! :)

Wes

MattAU

12:17 am on Oct 30, 2008 (gmt 0)

10+ Year Member



Check out [php.net...]

Then search google and you'll find heaps of examples.

wfernley

12:35 am on Oct 30, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Sounds good! Thanks for the link. I'm assuming I will incorporate cURL into AJAX?
 

Featured Threads

Hot Threads This Week

Hot Threads This Month