Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Google Index robots.txt


phranque - 10:31 pm on Aug 24, 2012 (gmt 0)


i believe most well-behaved robots will use the exclusion that is most specific to the user agent and if there is more than one relevant exclusion the bot will use the most restrictive.

i would assume in general that means "noindex, nofollow" for bots that support X-Robots-Tag.

however googlebot should see the more specific "googlebot" exclusion and go with "index, follow"


Thread source:: http://www.webmasterworld.com/robots_txt/4487907.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com