Msg#: 3109336 posted 3:38 am on Oct 6, 2006 (gmt 0)
Why Google database show robots.txt file?
That sure seems like an obvious question to me too. It seems like it would be simple enough for Google to implement. Is anyone really interested in seeing the contents of a robots.txt file in their SE results?
Msg#: 3109336 posted 4:00 am on Oct 6, 2006 (gmt 0)
Number 4 is BT's robots.txt blog :) number 5 is google's own robots txt file
Webmasterworld: Robots.txt Brett Tabke experiments with writing a weblog in a text file usually read only by robots. Trenchant commentary on the world of search engine marketing. www.webmasterworld.com/robots.txt - 2k - Cached - Similar pages