homepage Welcome to WebmasterWorld Guest from 54.227.171.163
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

    
Is it possible to scrape Google's results - legitimately?
Stevil

5+ Year Member



 
Msg#: 4287278 posted 10:47 pm on Mar 22, 2011 (gmt 0)

I'm currently working on (or trying to at least) a system which stores positions of my sites for certain keywords over a period of time. I've looked at google APIs and there a few ways which work, but all of them have a clause saying you can't do automated searches and it all has to be done via human interaction.

Is there any legit way to be able to get these results (even if it's paid) and not require human action?

There are several ways I'd be able to work around google's systems, but I'm looking to do everything above board... I have a feeling it might not be possible though!

Thanks in advance for any info people might have to offer!

 

tedster

WebmasterWorld Senior Member tedster us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 4287278 posted 1:53 pm on Mar 25, 2011 (gmt 0)

If you are asking about having Google's public blessing to scrape search results from google.com, I think you've already got the answer. The last time we had a thread about this was about a year ago - see Crawling Google - How can it be done? [webmasterworld.com]

Hisoka



 
Msg#: 4287278 posted 3:58 pm on Mar 25, 2011 (gmt 0)

There's really no way to do it legitimately without Google's permission.

Of course, if you're looking for shady ways, you have to use over 100 proxies, get rid of cookies after every visit, and make sure you don't hit it too often in a short period of time. It's very possible to scrape Google 24/7 if you have lots of proxies in your disposal, and you change your keywords often

deadsea

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 4287278 posted 7:03 pm on Mar 25, 2011 (gmt 0)

Google webmaster tools now gives you bulk data an what your site is ranking for. Go to webmaster tools -> "your site on the web" -> "search queries" -> "download this table". I get a spreadsheet with thousands of search phrases that includes average rank. You can even use the "Filters..." to restrict it to a specific country, search type, or substring match.

If you were willing to download this once a week, you could keep track over time pretty easily. I don't know if there is a programmatic way to log in and get this data automatically.

The rank data looks pretty good to me, but it does have a couple problems:
1. It isn't accounting for SEM ads or Google placements (map, images, currency, etc) that show up above the organic results.
2. We rank #1, #2, AND #3 for our brand name. It reports the average rank as 2.1. It would be far more useful for them to report average rank for my best slot (which in this case would be very close to 1).

tedster

WebmasterWorld Senior Member tedster us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 4287278 posted 8:20 pm on Mar 25, 2011 (gmt 0)

The WebmasterTools API [code.google.com] may offer some help.

Stevil

5+ Year Member



 
Msg#: 4287278 posted 2:46 am on Mar 28, 2011 (gmt 0)

Hisoka, you say it's not possible without Google's permission. Though for such small usage I know I'd never be able to get Google's permission, just curious as to if anyone has ever tried or know of anyone that has been given permission though?

I have got a dodgy version which does work, but don't really want to go down that path.

I've been trying to avoid using the web master tools API. I've noticed several of the issues that dead sea came across, plus it's more effort on my behalf. I may just have to put the idea on the back burner for awhile.

Thanks for the help everyone =)

incrediBILL

WebmasterWorld Administrator incredibill us a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



 
Msg#: 4287278 posted 2:58 am on Mar 28, 2011 (gmt 0)

Google scrapes us WITHOUT PERMISSION so grab a box of proxies and go at it IMO.

Pot calling kettle black, blah, etc.

See if 80legs would sell their botnet of crawlers with a custom user agent, would be epic.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved