Page is a not externally linkable
coffeelover - 7:41 pm on Apr 14, 2003 (gmt 0)
Do I need a robots.txt file to not scare off some spiders? Here is the sole log entry: No other entries. Is this normal? Will it be back?
I was hit by the Fast webcrawler today, but it only looked at my non-existant robots.txt
cr036r01-2.sac2.fastsearch.net - - [14/Apr/2003:14:55:30 -0400] "GET /robots.txt HTTP/1.0" 404 278 "-" "FAST-WebCrawler/3.8 (atw-crawler at fast dot no; [fast.no...]