Welcome to WebmasterWorld Guest from 50.19.156.19

Forum Moderators: open

Message Too Old, No Replies

How do I Block Fast

     

Elgoog

12:31 am on Dec 10, 2002 (gmt 0)



This is what I'm using in my .htaccess file now:

ErrorDocument 403 /var/www/html/Forbidden.html

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^FAST-WebCrawler/3\.6\ Agent$ [OR]
RewriteCond %{HTTP_USER_AGENT} FAST-WebCrawler\ Agent$ [OR]
RewriteCond %{HTTP_USER_AGENT} DTS\ Agent$ [NC,OR]
RewriteCond %{REMOTE_ADDR} ^218\.5\.77\.71$
RewriteRule .* - [F]

This is the whole string reported:

HTTP/1.0" 200 35838 "-" "FAST-WebCrawler/3.6 (atw-crawler at fast dot no; [fast.no...]

Elgoog

12:34 am on Dec 10, 2002 (gmt 0)



This is what I have in my robots.txt file. I just can't afford the upstream and plus they have been crawling for months and have never placed me in the results so I don't want them to crawl my sites.

User-agent: Googlebot
Disallow:
User-agent: *
Disallow: /
Disallow: admin.php
Disallow: config.php
Disallow: /admin/
Disallow: /images/
Disallow: /includes/
Disallow: /themes/
Disallow: /blocks/
Disallow: /modules/
Disallow: /language/
Disallow: /netjuke/

jdMorgan

12:43 am on Dec 10, 2002 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member



elgoog,

Welcome to WebmasterWorld [webmasterworld.com]!

You need a blank line between robots.txt records, as so:


User-agent: Googlebot
Disallow:

User-agent: *
Disallow: /
Disallow: /admin.php
Disallow: /config.php
Disallow: /admin/
... more ...
Disallow: /netjuke/

You can validate your robots.txt here [searchengineworld.com].

<added>Fast Web Crawler Information page is here [fastsearch.com].</added>

HTH,
Jim