Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

Help Please - FAST Crawler

10:13 am on Feb 27, 2003 (gmt 0)

New User

10+ Year Member

joined:Feb 27, 2003
votes: 0


I launched my new website couple of weeks back. When I checked my weblog I found the following record (just one single record)

---------------------------------------------------------- - - [25/Feb/2003:03:44:17 -0500] "GET /robots.txt HTTP/1.0" 404 283 "-" "FAST-WebCrawler/3.6 (atw-crawler at fast dot no; [fast.no...]

Does it mean that the FAST webcrawler (alltheweb.com) has spidered my website? if so, how long will take for my site to get indexed in FAST.

I don't have any robots.txt in my website.

Pls clarify.



10:16 am on Feb 27, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member heini is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Jan 31, 2001
votes: 0

Hi bashyam - welcome to WebmasterWorld

What that means is Fast knows about your site. The crawler has made a first visit, checking, correctly, for robots txt. The crawler now knows it is allowed to crawl your site. Which will eventually happen.
It does however not mean your site is to appear in the index without further crawling.


Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members