Page is a not externally linkable
- Google
-- Google SEO News and Discussion
---- Googlebot getting caught in robots.txt spider trap


goodroi - 12:30 pm on Aug 1, 2011 (gmt 0)


You might want to double check your claim. You may be correct but based on prior experience almost all of the people that have made this claim to me were mistaken.

Here are common issues I have come across:

1) Robots.txt was not correctly setup
2) Robots.txt is correct but was uploaded while Google was already crawling the content
3) Robots.txt is correct but placed in the wrong location
4) Someone is trying to scrape content & is faking the user agent to be googlebot (check the ip)

On rare occasions I have seen Google make a mistake. Considering they crawl billions of pages the occasional glitch is to be expected. If you have confidential information don't put it online. If you need to put sensitive information online use htaccess to further secure it.


Thread source:: http://www.webmasterworld.com/google/4346138.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com