johnd - 8:43 am on Jun 17, 2002 (gmt 0)

there seems to be no one answer to the question of robots.txt

it wouldn't be logical if not having one would be interpreted as disallow all, because a large number of web sites would never get a chance to be spidered (people who never heard of robots.txt - personal home pages etc would never make it to the web). it doesn't mean it's not true on the other hand.