Welcome to WebmasterWorld Guest from 54.167.40.25

Forum Moderators: open

Message Too Old, No Replies

Help Please - FAST Crawler

     
10:13 am on Feb 27, 2003 (gmt 0)

New User

10+ Year Member

joined:Feb 27, 2003
posts:17
votes: 0


Hi,

I launched my new website couple of weeks back. When I checked my weblog I found the following record (just one single record)

----------------------------------------------------------
66.77.73.58 - - [25/Feb/2003:03:44:17 -0500] "GET /robots.txt HTTP/1.0" 404 283 "-" "FAST-WebCrawler/3.6 (atw-crawler at fast dot no; [fast.no...]
----------------------------------------------------------

Does it mean that the FAST webcrawler (alltheweb.com) has spidered my website? if so, how long will take for my site to get indexed in FAST.

I don't have any robots.txt in my website.

Pls clarify.

Thanks,

Balaji.

10:16 am on Feb 27, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member heini is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Jan 31, 2001
posts:4404
votes: 0


Hi bashyam - welcome to WebmasterWorld

What that means is Fast knows about your site. The crawler has made a first visit, checking, correctly, for robots txt. The crawler now knows it is allowed to crawl your site. Which will eventually happen.
It does however not mean your site is to appear in the index without further crawling.