Welcome to WebmasterWorld Guest from 23.22.17.192

Forum Moderators: goodroi

Message Too Old, No Replies

Google's Nonstandard robots.txt is Not Valid

Googles robots.txt Does not validate

   
4:33 pm on Aug 6, 2005 (gmt 0)

5+ Year Member



Now, check out google's robots.txt.
I thought it was pretty clear to everybody that the Allow: tag was not correct...
If you try to validate [searchengineworld.com] www.google.com/robots.txt, you simply get an error that google uses the Allow: tag in its robots.txt!

Does this mean that googl'es agents actually use the Allow: tag?

4:22 pm on Aug 7, 2005 (gmt 0)

WebmasterWorld Administrator jatar_k is a WebmasterWorld Top Contributor of All Time 10+ Year Member



it isn't a supported tag, doesn't mean they can't use it
10:24 pm on Aug 7, 2005 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member



Well obviously they can and have used it, however they risk having other bots/spiders getting confused and disregarding all their rules.
1:35 pm on Aug 17, 2005 (gmt 0)

5+ Year Member



a bit off-topic, but I see /search listed in google's robots.txt and if you look at these results: [search.msn.com...] you'll see a google search page listed there.
which means search.msn has indexed a disallowed url. It's funny to see a google result page listed in a microsoft one :)

The msnbot has probably indexed this link found on several other websites...

1:54 pm on Aug 17, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



you'll see a google search page listed there.

Its a link that must have been found on one of the crawled pages elsewhere -- robots.txt only regulates which pages should NOT be retrieved, not which URLs should not ever be used to link to the site.

1:59 pm on Aug 17, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



> which means search.msn has indexed a disallowed url.

I think your conclusion is wrong - MSN has not indexed the page, only the URL. Google will also show URL only results, even if they're banned via robots.txt