Page is a not externally linkable
Fizzy - 9:45 pm on Nov 13, 2003 (gmt 0)
I have been getting regular visits from atw but for some reason it comes in, reads my robots.txt and then immediately leaves, like so: 66.77.73.89[06/Nov/2003:20:30:27GET / HTTP/1.020021224-FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp) All I have in my robots.txt file is the following: # User-agent: TurnitinBot User-agent: NPBot User-agent: Zao User-agent: ia_archiver User-agent: baiduspider Am I doing something wrong?
Hi all,
66.77.73.89[07/Nov/2003:21:12:45GET / HTTP/1.020021039-FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)
66.77.73.89[08/Nov/2003:15:30:23GET /robots.txt HTTP/1.0200332-FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)
66.77.73.89[09/Nov/2003:09:56:00GET / HTTP/1.020021039-FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)
66.77.73.89[10/Nov/2003:11:05:59GET / HTTP/1.020021039-FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)
66.77.73.89[10/Nov/2003:15:55:12GET /robots.txt HTTP/1.0200370-FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)
66.77.73.89[11/Nov/2003:11:45:15GET / HTTP/1.020021039-FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)
66.77.73.89[12/Nov/2003:12:59:16GET / HTTP/1.020021039-FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)
66.77.73.89[12/Nov/2003:16:42:49GET /robots.txt HTTP/1.0200370-FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)
# This restricts access to only known and registered robots.
#
User-agent: *
Disallow: /cgi-bin/
Disallow: /yabbse/
Disallow: /
Disallow: /yabbse/
Disallow: /
Disallow: /
I have been waiting for ATW for months now but it just isn't picking up.