Forum Moderators: open
No, robots.txt only works on robots that choose to read it and cooperate.
You will need to block the IP address either in your configuration or by other means, such as "black-holing" that IP address at the firewall.
The methods available to you depend on what server your site is hosted on, and you didn't say...
Jim
Example
<Files .htaccess>
deny from all
</Files>
deny from 192.168.1.1
The first three lines prevent anyone from viewing your .htaccess file. The fourth line will deny the individual IP from your entire domain. You can also modify it to read:
deny from 192.168.1.
Which would deny all IPs in the range of 192.168.1.1 to 192.168.1.254
Here are several methods to do it:
------
<Files .htaccess>
deny from all
</Files>
<Files *>
deny from 192.168.1.1
</Files>
SetEnvIf Remote_Addr ^192\.168\.0\.1$ banit
SetEnvIf Remote_Addr ^192\.168\.0\.8$ banit
SetEnvIf Remote_Addr ^192\.168\.0\.3$ banit
#
SetEnvIf Request_URI "^(/403.*\.html¦/robots\.txt)$" allowit
<Files *>
order deny,allow
deny from env=banit
allow from env=allowit
</Files>
RewriteCond %{REMOTE_ADDR} ^192\.168\.0\.1$ [OR]
RewriteCond %{REMOTE_ADDR} ^192\.168\.0\.8$ [OR]
RewriteCond %{REMOTE_ADDR} ^192\.168\.0\.3$
RewriteRule !^(robots\.txt¦403.*\.html)$ - [F]
The IP addresses shown are simply placeholders - there are more efficient ways to ban IP address ranges.
HTH,
Jim
<Files .htaccess>
deny from all
</Files>
Assuming your server is a relatively recent version of Apache (last year or two), this should not be necessary in your individual .htaccess as the Apache config file should already include it. Or more specifically, it says
<Files ~ "^\.ht">
Order allow,deny
Deny from all
Satisfy All
</Files>
which would also prevent people from snooping in .htpasswd files and etc.