Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Google's Current Specifications for Robots Directives


phranque - 9:00 am on Nov 26, 2010 (gmt 0)


I've been referencing RFC 1945 for years and some folks have argued with me on how many hops Googlebot will follow.

it should be noted that the statement you quoted is specifically referring to googlebot requests of the robots.txt file and in typical google fashion is neither specific about how many more or less than 5 hops googlebot will endure while requesting robots.txt nor about how many hops are acceptable for any other urls requested by googlebot.


Thread source:: http://www.webmasterworld.com/robots_txt/4234719.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com