Page is a not externally linkable
- Google
-- Google SEO News and Discussion
---- Why Google Might "Ignore" a robots.txt Disallow Rule


mcavic - 10:20 pm on Aug 15, 2006 (gmt 0)


So ... according to a close reading of the specification, the rules for a specific user agent _entirely override_ the "User-agent: *" rules. Therefore, any rule under "User-agent: *" that should also be applied to googlebot must be repeated under "User-agent: googlebot."

Correct.


Thread source:: http://www.webmasterworld.com/google/3044757.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com