Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

ATW reads robots.txt then leaves



9:45 pm on Nov 13, 2003 (gmt 0)

10+ Year Member

Hi all,

I have been getting regular visits from atw but for some reason it comes in, reads my robots.txt and then immediately leaves, like so:[06/Nov/2003:20:30:27GET / HTTP/1.020021224-FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)[07/Nov/2003:21:12:45GET / HTTP/1.020021039-FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)[08/Nov/2003:15:30:23GET /robots.txt HTTP/1.0200332-FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)[09/Nov/2003:09:56:00GET / HTTP/1.020021039-FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)[10/Nov/2003:11:05:59GET / HTTP/1.020021039-FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)[10/Nov/2003:15:55:12GET /robots.txt HTTP/1.0200370-FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)[11/Nov/2003:11:45:15GET / HTTP/1.020021039-FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)[12/Nov/2003:12:59:16GET / HTTP/1.020021039-FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)[12/Nov/2003:16:42:49GET /robots.txt HTTP/1.0200370-FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)

All I have in my robots.txt file is the following:

# This restricts access to only known and registered robots.
User-agent: *
Disallow: /cgi-bin/

User-agent: TurnitinBot
Disallow: /yabbse/

User-agent: NPBot
Disallow: /

User-agent: Zao
Disallow: /yabbse/

User-agent: ia_archiver
Disallow: /

User-agent: baiduspider
Disallow: /

Am I doing something wrong?
I have been waiting for ATW for months now but it just isn't picking up.


9:50 pm on Nov 13, 2003 (gmt 0)

WebmasterWorld Senior Member macguru is a WebmasterWorld Top Contributor of All Time 10+ Year Member

Hi Fizzy,

The bot is teasing you. ;)

Get more inbound links, else pull the plastic.


11:04 pm on Nov 13, 2003 (gmt 0)

10+ Year Member

Hi Mac,

Well I already have over 400 inbound links according to ATW so I'm not sure how many I would need for them to decide to spider the site.

I have waited patiently for many months now and have yet to see them go through the site. The pages are shtml so I would have thought that they are spider fodder, it's not even trying to spider them though.
Could this be because ATW is coming in on the inbound links and isn't bothering to actually visit my site to spider it? Could it just be "passing through"?


11:48 pm on Nov 13, 2003 (gmt 0)

10+ Year Member

Fizzy, from what I have seen of ATW it is very slow. On a mid-sized site of mine it is generally about 5-6 months behind in indexing the content when compared to Google.

I see lots of requests for robots.txt and then nothing else, similar to what you describe. Some indexing must occur occasionally though, just not very often.

I like the ATW interface but my log files tell me it sends me very few visitors.


11:01 am on Nov 14, 2003 (gmt 0)

WebmasterWorld Senior Member heini is a WebmasterWorld Top Contributor of All Time 10+ Year Member

From what I see in my logs the FirstPage crawler has been in charge for index page checking. Deep indexing was from FAST-WebCrawler/3.8.
That one, WebCrawler/3.8./Fresh, really lives up to it's name, it comes daily.

Fizzy, there have been several reports lately of sites where only the robots txt gets checked but nothing indexed, even for well linked sites.
Frankly I don't know what the problem is. It might have to do with the behind the scenes working at OV.
We have heard the frontend serving Altavista and ATW has been merged. The bigger question is what about the backend?


1:12 pm on Nov 14, 2003 (gmt 0)

10+ Year Member

heini, sounds like you are getting some nice regular visits :).

I contacted the ATW support people by email when I first noticed this robots.txt thing, and they asked for a log file snippet so they could see what was happening. Never heard anything back though.


10:25 am on Nov 29, 2003 (gmt 0)

10+ Year Member

Thanks for the replies everybody.

I'll keep watching and see if anything changes, nothing has so far though :(


11:25 pm on Dec 7, 2003 (gmt 0)

10+ Year Member

Hi all,

I promised an update if I got it as I was worried about being missed out.

FAST-WebCrawler/3.8 - atw-crawler at fast dot no (117 pages and counting)

Thanks again to you all for your continued kind and helpful advice.


Featured Threads

Hot Threads This Week

Hot Threads This Month