Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

Help Please - FAST Crawler



10:13 am on Feb 27, 2003 (gmt 0)

10+ Year Member


I launched my new website couple of weeks back. When I checked my weblog I found the following record (just one single record)

---------------------------------------------------------- - - [25/Feb/2003:03:44:17 -0500] "GET /robots.txt HTTP/1.0" 404 283 "-" "FAST-WebCrawler/3.6 (atw-crawler at fast dot no; [fast.no...]

Does it mean that the FAST webcrawler (alltheweb.com) has spidered my website? if so, how long will take for my site to get indexed in FAST.

I don't have any robots.txt in my website.

Pls clarify.




10:16 am on Feb 27, 2003 (gmt 0)

WebmasterWorld Senior Member heini is a WebmasterWorld Top Contributor of All Time 10+ Year Member

Hi bashyam - welcome to WebmasterWorld

What that means is Fast knows about your site. The crawler has made a first visit, checking, correctly, for robots txt. The crawler now knows it is allowed to crawl your site. Which will eventually happen.
It does however not mean your site is to appear in the index without further crawling.


Featured Threads

Hot Threads This Week

Hot Threads This Month