Page is a not externally linkable
- Yahoo
-- Deprecated - Altavista, Alltheweb.com
---- ATW reads robots.txt then leaves


Fizzy - 9:45 pm on Nov 13, 2003 (gmt 0)


Hi all,

I have been getting regular visits from atw but for some reason it comes in, reads my robots.txt and then immediately leaves, like so:

66.77.73.89[06/Nov/2003:20:30:27GET / HTTP/1.020021224-FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)
66.77.73.89[07/Nov/2003:21:12:45GET / HTTP/1.020021039-FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)
66.77.73.89[08/Nov/2003:15:30:23GET /robots.txt HTTP/1.0200332-FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)
66.77.73.89[09/Nov/2003:09:56:00GET / HTTP/1.020021039-FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)
66.77.73.89[10/Nov/2003:11:05:59GET / HTTP/1.020021039-FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)
66.77.73.89[10/Nov/2003:15:55:12GET /robots.txt HTTP/1.0200370-FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)
66.77.73.89[11/Nov/2003:11:45:15GET / HTTP/1.020021039-FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)
66.77.73.89[12/Nov/2003:12:59:16GET / HTTP/1.020021039-FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)
66.77.73.89[12/Nov/2003:16:42:49GET /robots.txt HTTP/1.0200370-FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)

All I have in my robots.txt file is the following:

#
# This restricts access to only known and registered robots.
#
User-agent: *
Disallow: /cgi-bin/

User-agent: TurnitinBot
Disallow: /yabbse/

User-agent: NPBot
Disallow: /

User-agent: Zao
Disallow: /yabbse/

User-agent: ia_archiver
Disallow: /

User-agent: baiduspider
Disallow: /

Am I doing something wrong?
I have been waiting for ATW for months now but it just isn't picking up.


Thread source:: http://www.webmasterworld.com/altavista/972.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com