- - Google
- -- Google News Archive
- ---- Google.com now has no robots.txt?
dmorison - 7:07 am on Sep 2, 2003 (gmt 0)
|Surely Google have the ability to control what their own bot indexes on their own site withot having to specify in robots.txt? |
Of course; but given that Google need to use a robots.txt to block other crawlers there is no reason why it should not be the only mechanism in place to stop their own.
Thread source: http://www.webmasterworld.com/google_archive/16587.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com