Crawling Google - How can it be done?

[vincevincevince]

What legitimate methods are there for automated crawling of Google search results, paid or unpaid? I know direct crawling is prohibited by their TOS, yet I cannot find an alternative method.

[tedster]

You ask for "legitimate methods" but acknowledge that automated requests are against the Google TOS. That's your answer, I'm afraid. Any automated scraping of Google results does break their TOS.

[whoisgregg]

Depending on what you are trying to accomplish, perhaps the Google AJAX Search API [code.google.com] could be helpful?

[vincevincevince]

Thanks whoisgregg; it is a step in the right direction at least. Sad thing is I'm quite happy to pay Google for the results, they just don't seem to have a programme to allow me to do so.

[triggerfinger]

You will probably need a developer to design such a program. I know of a developer who made one such program for a client of his, I don't know much about it, except that you need to use different IPs and different times so as not to trigger the algo's spam warning.

[claus]

Just a brief comment: AFAIK, the API SERPS and the WWW SERPS tend to differ.

But then, the WWW SERPS tend to differ a bit anyway, based on geolocation, search history, quality of pizza's in GOOG HQ and whatnot.

:-)

[Brett_Tabke]

Remember, google has several "partners" that present essentially the same serps...

[vincevincevince]

Thanks triggerfinger; I am aware of the more sneaky ways to do it; but obviously would rather if at all possible obtain things in a permissable way.

Brett, thanks for that hint! I will be following up on that in detail this week.