homepage Welcome to WebmasterWorld Guest from 23.20.34.25
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld
Home / Forums Index / Code, Content, and Presentation / PHP Server Side Scripting
Forum Library, Charter, Moderators: coopster & jatar k

PHP Server Side Scripting Forum

    
Grabbing remote source code
wbsmperry




msg:3703742
 7:20 pm on Jul 21, 2008 (gmt 0)

Trying to read the source code from a website, find data in the source code, and save the data I am looking for, for use later. But I am getting an error that my browser is not supported. Is there anyway to get past this?

 

eelixduppy




msg:3703757
 7:31 pm on Jul 21, 2008 (gmt 0)

Welcome to WebmasterWorld! :)

You are getting this error from the URL you are requesting? Sounds like they are checking user-agents and displaying content accordingly.

wbsmperry




msg:3703762
 7:33 pm on Jul 21, 2008 (gmt 0)

yes, is there anyway to "fake" the user agent though? This is driving me crazy, lol

eelixduppy




msg:3703784
 7:51 pm on Jul 21, 2008 (gmt 0)

You'd have to use cURL to change the user-agent. It looks like this:

[url=http://www.php.net/curl-setopt]curl_setopt[/url]($ch, CURLOPT_USERAGENT, $useragent);

Read up on cURL at the documentation: [php.net...]

wbsmperry




msg:3703788
 7:52 pm on Jul 21, 2008 (gmt 0)

Thanks for the info, and the Welcome :) I've been a reader here for quite some time, but decided it was time to register and get asking :) Thanks again, I will give this a go :)

wbsmperry




msg:3703817
 8:07 pm on Jul 21, 2008 (gmt 0)

alright, Is there any chance you could give me a snippet of code that would set a user agent, and read the source code from say google.com?

wbsmperry




msg:3703837
 8:20 pm on Jul 21, 2008 (gmt 0)

I got it :-P

but its still not working quite right. :( it will load pages like google.com fine, but when I try to load a page from facebook, it either gives me a blank page, or brings me to the login page. :-(

bcolflesh




msg:3703857
 8:28 pm on Jul 21, 2008 (gmt 0)

You can use cURL to login into password protected sites, like Facebook - you need to read the cURL docs, as mentioned above.

wbsmperry




msg:3703863
 8:32 pm on Jul 21, 2008 (gmt 0)

lol, thanks bcolflesh. Sorry for being a pain, just impatient, have been trying to get this working for sooooo long now!

bcolflesh




msg:3703864
 8:34 pm on Jul 21, 2008 (gmt 0)

Try a G search for:

php authenticating with curl

eelixduppy




msg:3703865
 8:34 pm on Jul 21, 2008 (gmt 0)

If you are looking for quick, there may be scripts already written for logging into facebook. A quick Google search or a search at [phpclasses.org...] might be in order. :)

[edit]
beaten to it ;)

wbsmperry




msg:3703873
 8:42 pm on Jul 21, 2008 (gmt 0)

thanks again folks, I am just at a loss as to why there would be nothing returned at all from facebook?

bcolflesh




msg:3703890
 8:59 pm on Jul 21, 2008 (gmt 0)

thanks again folks, I am just at a loss as to why there would be nothing returned at all from facebook?

1. The session you initiated with cURL isn't logged in.
2. Facebook is checking for a cookie to be set that you haven't set with cURL
3. The user-agent you are sending with cURL doesn't fit a pattern Facebook allows
...

The list goes on - most folks don't want Joe Blow to scrape them and don't make it easy.

wbsmperry




msg:3703892
 9:00 pm on Jul 21, 2008 (gmt 0)

Great in, thanks alot, i will see what i can dig up

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Code, Content, and Presentation / PHP Server Side Scripting
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved