Page is a not externally linkable
- Google
-- Google SEO News and Discussion
---- Why Google Might "Ignore" a robots.txt Disallow Rule


mcavic - 7:07 pm on Aug 13, 2006 (gmt 0)


What appears to happen is that as soon as you add a User-agent: Googlebot section, that Google starts indexing URLs that are not mentioned in that section, even if they are mentioned in the User-agent: * section

Yes, that's how it should be, because you might want to allow Google more access than all other bots.


Thread source:: http://www.webmasterworld.com/google/3044757.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com