| 4:49 am on Dec 5, 2007 (gmt 0)|
There are actually two PTR records for addresses in that range; If your server checks only the first (which appears to be a CNAME), then it may fail rDNS verification. The second record points to hotmail.com
Based on this and what I've seen of their robots.txt and on-page robots meta-tag handling, I have to agree with your assessment.
[edited by: jdMorgan at 4:49 am (utc) on Dec. 5, 2007]
| 5:30 am on Dec 6, 2007 (gmt 0)|
Now you know why I use a combination of methods.
Cached reverse DNS lookups for a quick test for bot validity and if that fails I check to see if it at least belongs to a range owned by the company associated with the bot and do this for Google, Yahoo, Microsoft, Ask, Gigablast, etc.
[edited by: incrediBILL at 5:30 am (utc) on Dec. 6, 2007]
| 6:56 am on Dec 6, 2007 (gmt 0)|
Thanks Shurik for detecting this configuration issue on few servers, we will fix that.
| 2:15 pm on Dec 6, 2007 (gmt 0)|
Shurik wrote: This breaks reverse DNS lookup on my site (and many other sites) with all the nasty consequences.
Shurik: Excuse my lack of knowledge in this area, but does this have something to do with Slow Crawling, No Indexing or NO RANKING and NOT SHOWING in serps?
My MSN traffic stopped cold Oct 1st at 10 am Pacific Time when they switched on the new Algo.
Will this "fix" that MSNDude speaks of, address the issue and return the many lost sites to the serps?
Many of us are at a loss as to why we are not in the serps with this new Algo.
Thanks in advance for a post.
| 10:32 pm on Mar 10, 2008 (gmt 0)|
188.8.131.52 - - [29/Feb/2008:02:46:46 +0100] "GET / HTTP/1.1" 403 *** "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
184.108.40.206 - - [07/Mar/2008:03:53:54 +0100] "GET / HTTP/1.1" 403 *** "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
220.127.116.11 - - [08/Mar/2008:14:05:33 +0100] "GET /robots.txt HTTP/1.1" 200 *** "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
18.104.22.168 = bl1sch2033920.phx.gbl
22.214.171.124 = bl1sch2042520.phx.gbl
126.96.36.199 = by1sch4030220.phx.gbl
| 4:03 pm on Mar 11, 2008 (gmt 0)|
Four month passed and the problem is still not fixed.
Shame on Microsoft.