Forum Moderators: open
I've been getting these "crawls" for more than a week.
They are very slow with around 10-20 scattered through-out a 24-hour period.
I haven't a clue if it's a bot or a private user.
Never any images, just page and no robots.
I denied it to the class C range.
We had something similar last month. It was coming from a valid MSN range - hotmail - and claiming to be msnbot, which of course it was not. It started off slowly but quickly became fairly aggressive.
See: [webmasterworld.com...]
In the end we ended up validating msnbot ranges to stop spoofer(s), and after being 403'd a few 100 times they have not since returned.
# msnbot 207.46.0.0 - 207.46.255.255, 207.68.128.0 - 207.68.207.255, 65.52.0.0 - 65.55.255.255
RewriteCond %{HTTP_USER_AGENT} ^msnbot [NC]
RewriteCond %{REMOTE_ADDR}!^207\.46\.
RewriteCond %{REMOTE_ADDR}!^207\.68\.(12[89]¦1[3-9][0-9]¦20[0-7])\.
RewriteCond %{REMOTE_ADDR}!^65\.5[2-5]\.
RewriteRule .* - [F]
GG
# msnbot 207.46.0.0 - 207.46.255.255, 207.68.128.0 - 207.68.207.255, 65.52.0.0 - 65.55.255.255
RewriteCond %{HTTP_USER_AGENT} ^msnbot [NC]
RewriteCond %{REMOTE_ADDR}!^207\.46\.
RewriteCond %{REMOTE_ADDR}!^207\.68\.(12[89]¦1[3-9][0-9]¦20[0-7])\.
RewriteCond %{REMOTE_ADDR}!^65\.5[2-5]\.
RewriteRule .* - [F]
That's plenty of excess in lines when
deny from 65.54.158.
solves the problem ;)
Although I stil have the following in my htaccess from 3-4 years ago:
RewriteCond %{HTTP_USER_AGENT} ^Microsoft
RewriteCond %{REMOTE_ADDR}!^131\.107\.
RewriteRule .* - [F]
end of quote
There are multiple lengthy threads in the "would be archives" from when the MSN bot began anonymously.
[webmasterworld.com...]
Don
At least for now ;)
