- - Google
- -- Google SEO News and Discussion
- ---- Amazon S3 Forum on Google - Appspot Clone Has the Top 2 Results
tedster - 10:23 pm on Feb 14, 2011 (gmt 0)
Here are some more pieces of the puzzle. I noticed that the Amazon url is a URL-only listing. That clued me in to check the robots.txt file at Amazon.
In fact, the robots.txt file specifically excludes googlebot (and only googlebot) from crawling the forum.
|User-agent: * |
Thread source: http://www.webmasterworld.com/google/4266385.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com