Welcome to WebmasterWorld Guest from 54.146.174.220

Forum Moderators: goodroi

Google's Nonstandard robots.txt is Not Valid

Googles robots.txt Does not validate

   
4:33 pm on Aug 6, 2005 (gmt 0)

5+ Year Member



Now, check out google's robots.txt.
I thought it was pretty clear to everybody that the Allow: tag was not correct...
If you try to validate [searchengineworld.com] www.google.com/robots.txt, you simply get an error that google uses the Allow: tag in its robots.txt!

Does this mean that googl'es agents actually use the Allow: tag?

4:22 pm on Aug 7, 2005 (gmt 0)

WebmasterWorld Administrator jatar_k is a WebmasterWorld Top Contributor of All Time 10+ Year Member



it isn't a supported tag, doesn't mean they can't use it
10:24 pm on Aug 7, 2005 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member



Well obviously they can and have used it, however they risk having other bots/spiders getting confused and disregarding all their rules.
1:35 pm on Aug 17, 2005 (gmt 0)

5+ Year Member



a bit off-topic, but I see /search listed in google's robots.txt and if you look at these results: [search.msn.com...] you'll see a google search page listed there.
which means search.msn has indexed a disallowed url. It's funny to see a google result page listed in a microsoft one :)

The msnbot has probably indexed this link found on several other websites...

1:54 pm on Aug 17, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



you'll see a google search page listed there.

Its a link that must have been found on one of the crawled pages elsewhere -- robots.txt only regulates which pages should NOT be retrieved, not which URLs should not ever be used to link to the site.

1:59 pm on Aug 17, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



> which means search.msn has indexed a disallowed url.

I think your conclusion is wrong - MSN has not indexed the page, only the URL. Google will also show URL only results, even if they're banned via robots.txt

 

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month