joined:Apr 19, 2007
Today i got hits from MSRBOT of microsoft and getting 404's. My log excerpts,
131.107.151.zzz - "GET /not-there-1.txt HTTP/1.0" 404 42 "-" "MSRBOT (http://research.microsoft.com/research/sv/msrbot/"
131.107.151.zzz - "GET /not-there-2.txt HTTP/1.0" 404 42 "-" "MSRBOT (http://research.microsoft.com/research/sv/msrbot/"
The last number in the ip range is 3 digit, don't know, why 3 x's change to these things!
There is some strange thing in its behaviour,
a) it is asking for non-existent page
b) When i tried to view the URL it supplied in UA string, i can't reach anywhere.
After google-ing i found one page here [research.microsoft.com].
This entry is from that page
Why is MSRBot trying to download incorrect links from my server? Or from a server that doesn't exist? Because MSRBot obtains the list of links to crawl by extracting them from documents on the web, there must be an incorrect link available on the web. To determine the location of this links, look at the referral field in your web server log.
Now, how i am suppose to look for the referer, if it is blank, is anybody's guess. ;)
i found this post
[webmasterworld.com] in WebmasterWorld.
Note, before they were crawling from above.net, ip - 209.249.11.x and now they are coming from ip owned by microsoft.
Since their behaviour is suspicious i banned them through .htaccess.
anybody else seeing this?