homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
Forum Library, Charter, Moderator: open

Deprecated - Altavista, Alltheweb.com Forum

Help Please - FAST Crawler

10+ Year Member

Msg#: 817 posted 10:13 am on Feb 27, 2003 (gmt 0)


I launched my new website couple of weeks back. When I checked my weblog I found the following record (just one single record)

---------------------------------------------------------- - - [25/Feb/2003:03:44:17 -0500] "GET /robots.txt HTTP/1.0" 404 283 "-" "FAST-WebCrawler/3.6 (atw-crawler at fast dot no; [fast.no...]

Does it mean that the FAST webcrawler (alltheweb.com) has spidered my website? if so, how long will take for my site to get indexed in FAST.

I don't have any robots.txt in my website.

Pls clarify.





WebmasterWorld Senior Member heini us a WebmasterWorld Top Contributor of All Time 10+ Year Member

Msg#: 817 posted 10:16 am on Feb 27, 2003 (gmt 0)

Hi bashyam - welcome to WebmasterWorld

What that means is Fast knows about your site. The crawler has made a first visit, checking, correctly, for robots txt. The crawler now knows it is allowed to crawl your site. Which will eventually happen.
It does however not mean your site is to appear in the index without further crawling.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved