Welcome to WebmasterWorld Guest from 18.104.22.168
Forum Moderators: open
Do I need a robots.txt file to not scare off some spiders?
Here is the sole log entry:
cr036r01-2.sac2.fastsearch.net - - [14/Apr/2003:14:55:30 -0400] "GET /robots.txt HTTP/1.0" 404 278 "-" "FAST-WebCrawler/3.8 (atw-crawler at fast dot no; [fast.no...]
No other entries.
Is this normal? Will it be back?