homepage Welcome to WebmasterWorld Guest from 23.20.196.179
register, login, search, subscribe, help, library, PubCon, announcements, recent posts, open posts,
Subscribe to WebmasterWorld
Visit PubCon.com
Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
Forum Library : Charter : Moderator: open

Deprecated - Altavista, Alltheweb.com Forum

    
How do I Block Fast
Elgoog




msg:223793		 12:31 am on Dec 10, 2002 (gmt 0)

This is what I'm using in my .htaccess file now:

ErrorDocument 403 /var/www/html/Forbidden.html

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^FAST-WebCrawler/3\.6\ Agent$ [OR]
RewriteCond %{HTTP_USER_AGENT} FAST-WebCrawler\ Agent$ [OR]
RewriteCond %{HTTP_USER_AGENT} DTS\ Agent$ [NC,OR]
RewriteCond %{REMOTE_ADDR} ^218\.5\.77\.71$
RewriteRule .* - [F]

This is the whole string reported:

HTTP/1.0" 200 35838 "-" "FAST-WebCrawler/3.6 (atw-crawler at fast dot no; [fast.no...]

 

Elgoog




msg:223794		 12:34 am on Dec 10, 2002 (gmt 0)

This is what I have in my robots.txt file. I just can't afford the upstream and plus they have been crawling for months and have never placed me in the results so I don't want them to crawl my sites.

User-agent: Googlebot
Disallow:
User-agent: *
Disallow: /
Disallow: admin.php
Disallow: config.php
Disallow: /admin/
Disallow: /images/
Disallow: /includes/
Disallow: /themes/
Disallow: /blocks/
Disallow: /modules/
Disallow: /language/
Disallow: /netjuke/

jdMorgan




msg:223795		 12:43 am on Dec 10, 2002 (gmt 0)

elgoog,

Welcome to WebmasterWorld [webmasterworld.com]!

You need a blank line between robots.txt records, as so:

User-agent: Googlebot
Disallow:

User-agent: *
Disallow: /
Disallow: /admin.php
Disallow: /config.php
Disallow: /admin/
... more ...
Disallow: /netjuke/

You can validate your robots.txt here [searchengineworld.com].

<added>Fast Web Crawler Information page is here [fastsearch.com].</added>

HTH,
Jim

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
WebmasterWorld ® and PubCon ® are a Registered Trademarks of Pubcon Inc.
© Pubcon Inc. 1996-2012 all rights reserved