homepage Welcome to WebmasterWorld Guest from 50.17.21.7
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Code, Content, and Presentation / PHP Server Side Scripting
Forum Library, Charter, Moderators: coopster & jatar k

PHP Server Side Scripting Forum

    
Curl Scraping Problem
Elric99




msg:4191365
 10:24 am on Aug 24, 2010 (gmt 0)

Hello,

My business uses a lot of CURL to get info from websites. (Not scraping in the sense of MFA websites).

The problem is that CURL scrapes the source code, but I only want to scrape what the user sees - after javascript has been rendered etc.

Is this possible? If anyone could point me in the right direction I'd really appreciate it.

Thanks

 

lostdreamer




msg:4191375
 10:57 am on Aug 24, 2010 (gmt 0)

For the rendering of javascript, cURL is not the way to go.
You'll need to automate a browser to open the page and render the entire DOM.

seleniumhq.org
watir.com

You could also use the crowbar proxy.
It will render the page in gecko browser and send the rendered DOM to you.

simile.mit.edu/wiki/Crowbar

Hope this helps

Elric99




msg:4191830
 8:32 am on Aug 25, 2010 (gmt 0)

Perfect, Lostdreamer thank you!

Tom

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Code, Content, and Presentation / PHP Server Side Scripting
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved