Page is a not externally linkable
- Microsoft
-- Bing Search Engine News
---- Bing ignoring robots.txt?


revrob - 8:50 am on Sep 29, 2012 (gmt 0)


Thank you for your reply.

At the moment I do want to send those IP ranges to robots.txt and see what happens. AFAICS there is one bingbot IP range that behaves itself, by requesting robots.txt and obeying it, for example 65.55.52.111
NetRange: 65.52.0.0 - 65.55.255.255
CIDR: 65.52.0.0/14
and that is the range I don't restrict.

The others seem to be uncontrollable. I will probably put an [F] rule to replace the redirect to robots in due course.

I'd still be grateful if someone could explain exactly the difference between a 302 redirect and an internal rewrite.
(BTW - I don't control my own web server, so only have access to .htaccess and robots.txt controls)

Once again - thank you.


Thread source:: http://www.webmasterworld.com/msn_microsoft_search/4441555.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com