Grabbing remote source code

[wbsmperry]

Trying to read the source code from a website, find data in the source code, and save the data I am looking for, for use later. But I am getting an error that my browser is not supported. Is there anyway to get past this?

[eelixduppy]

Welcome to WebmasterWorld! :)

You are getting this error from the URL you are requesting? Sounds like they are checking user-agents and displaying content accordingly.

[wbsmperry]

yes, is there anyway to "fake" the user agent though? This is driving me crazy, lol

[eelixduppy]

You'd have to use cURL to change the user-agent. It looks like this:

[url=http://www.php.net/curl-setopt]curl_setopt[/url]($ch, CURLOPT_USERAGENT, $useragent);

Read up on cURL at the documentation: [php.net...]

[wbsmperry]

Thanks for the info, and the Welcome :) I've been a reader here for quite some time, but decided it was time to register and get asking :) Thanks again, I will give this a go :)

[wbsmperry]

alright, Is there any chance you could give me a snippet of code that would set a user agent, and read the source code from say google.com?

[wbsmperry]

I got it :-P

but its still not working quite right. :( it will load pages like google.com fine, but when I try to load a page from facebook, it either gives me a blank page, or brings me to the login page. :-(

[bcolflesh]

You can use cURL to login into password protected sites, like Facebook - you need to read the cURL docs, as mentioned above.

[wbsmperry]

lol, thanks bcolflesh. Sorry for being a pain, just impatient, have been trying to get this working for sooooo long now!

[bcolflesh]

Try a G search for:

php authenticating with curl

[eelixduppy]

If you are looking for quick, there may be scripts already written for logging into facebook. A quick Google search or a search at [phpclasses.org...] might be in order. :)

[edit]
beaten to it ;)

[wbsmperry]

thanks again folks, I am just at a loss as to why there would be nothing returned at all from facebook?

[bcolflesh]

thanks again folks, I am just at a loss as to why there would be nothing returned at all from facebook?

1. The session you initiated with cURL isn't logged in.
2. Facebook is checking for a cookie to be set that you haven't set with cURL
3. The user-agent you are sending with cURL doesn't fit a pattern Facebook allows
...

The list goes on - most folks don't want Joe Blow to scrape them and don't make it easy.

[wbsmperry]

Great in, thanks alot, i will see what i can dig up