Automated searching via Google's AJAX API

...limits and conditions

[dwhite]

I posted an earlier version of this topic in the PHP forum, but this forum is now appropriate considering how Google has changed. (Also the question/s in this post have changed somewhat since last time).

Basically, I want to scrape Google's search results and extract particular data (I don't need particularly to display to the end user the original/modified search results).

I've looked at the FAQ and docs on Google's AJAX search API [code.google.com], and unfortunately, it doesn'y answer some basic questions. I'm hoping you guys may know a little more.

What I want to know is:

a: What's the maximum number of megabytes or queries of search I can automatically extract a day without annoying Google? With their old PHP SOAP system [code.google.com], it was 1000 queries a day.

b: Can you have more than 10 results per 'page' or query? I need 100, or preferably, all 1000 results at once.

c: Can you pay for further automated querying if you go past the limit?

d: For speed's sake, how many connections can I open at once to the Google servers? Obviously I'd love to open say 10 connections at once as this would speed up the search page downloads.

This is all for a really fun and potentially revolutionary project, but I need to know if it's even possible.

[Quadrille]

Google does not 'go a bundle' on automated searches, and may bar you from searching if you trigger a limiter.

You need to read the API TOS very carefully to see what you are allowed - I suspect what you seek is way beyond what they'll give.

[dwhite]

You're probably right. To cut a long story short, their old SOAP API actually did allow 1000 queries per day, but their new API doesn't allow any automated querying :( See this thread [webmasterworld.com] for details.