homepage Welcome to WebmasterWorld Guest from 72.44.48.122
register, login, search, subscribe, help, library, PubCon, announcements, recent posts, open posts,
Pubcon Platinum Sponsor
Visit PubCon.com
Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
Forum Library : Charter : Moderator: open

Deprecated - Altavista, Alltheweb.com Forum

    
Help Please - FAST Crawler
bashyam




msg:219976		 10:13 am on Feb 27, 2003 (gmt 0)

Hi,

I launched my new website couple of weeks back. When I checked my weblog I found the following record (just one single record)

----------------------------------------------------------
66.77.73.58 - - [25/Feb/2003:03:44:17 -0500] "GET /robots.txt HTTP/1.0" 404 283 "-" "FAST-WebCrawler/3.6 (atw-crawler at fast dot no; [fast.no...]
----------------------------------------------------------

Does it mean that the FAST webcrawler (alltheweb.com) has spidered my website? if so, how long will take for my site to get indexed in FAST.

I don't have any robots.txt in my website.

Pls clarify.

Thanks,

Balaji.

 

heini




msg:219977		 10:16 am on Feb 27, 2003 (gmt 0)

Hi bashyam - welcome to WebmasterWorld

What that means is Fast knows about your site. The crawler has made a first visit, checking, correctly, for robots txt. The crawler now knows it is allowed to crawl your site. Which will eventually happen.
It does however not mean your site is to appear in the index without further crawling.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
WebmasterWorld ® and PubCon ® are a Registered Trademarks of Pubcon Inc.
© Pubcon Inc. 1996-2012 all rights reserved