Forum Moderators: open
ErrorDocument 403 /var/www/html/Forbidden.html
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^FAST-WebCrawler/3\.6\ Agent$ [OR]
RewriteCond %{HTTP_USER_AGENT} FAST-WebCrawler\ Agent$ [OR]
RewriteCond %{HTTP_USER_AGENT} DTS\ Agent$ [NC,OR]
RewriteCond %{REMOTE_ADDR} ^218\.5\.77\.71$
RewriteRule .* - [F]
This is the whole string reported:
HTTP/1.0" 200 35838 "-" "FAST-WebCrawler/3.6 (atw-crawler at fast dot no; [fast.no...]
User-agent: Googlebot
Disallow:
User-agent: *
Disallow: /
Disallow: admin.php
Disallow: config.php
Disallow: /admin/
Disallow: /images/
Disallow: /includes/
Disallow: /themes/
Disallow: /blocks/
Disallow: /modules/
Disallow: /language/
Disallow: /netjuke/
Welcome to WebmasterWorld [webmasterworld.com]!
You need a blank line between robots.txt records, as so:
User-agent: Googlebot
Disallow:User-agent: *
Disallow: /
Disallow: /admin.php
Disallow: /config.php
Disallow: /admin/
... more ...
Disallow: /netjuke/
You can validate your robots.txt here [searchengineworld.com].
<added>Fast Web Crawler Information page is here [fastsearch.com].</added>
HTH,
Jim