[webmasterworld.com...]
And have caught some leachers, it's going well.
Though two recent entries are of the form:
The ip address ^207\.68\.146\.40$ has been banned on Thu Apr 14 09:41:29 2005
The associated user agent was msnbot/1.0 (+http://search.msn.com/msnbot.htm)
The ip address ^207\.68\.146\.47$ has been banned on Sat Apr 16 09:48:07 2005
The associated user agent was msnbot/1.0 (+http://search.msn.com/msnbot.htm)
The script works by adding a url to the robots.txt as a disallow. I added the disallow a week before implementing the script.
So is Microsoft really ignoring my robots.txt?
User-agent: *
Sorted the problem... doh..
