Welcome to WebmasterWorld Guest from 50.17.114.227

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies

Should I block msnbot-media

     
10:20 am on Apr 9, 2014 (gmt 0)

New User

5+ Year Member

joined:Jan 17, 2008
posts: 4
votes: 0


I've noticed it's blocked on this default list:
[github.com...]

Is this really recommended? could it hurt my indexing and ranking with bing?
5:54 pm on Apr 9, 2014 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:5803
votes: 64


I don't block msnbot-media. It is used for several purposes including the Bing Image Search. If you don't want your images indexed, disallow your image directories in robots.txt or use an X-Robots-Tag in your directory's htaccess.

I would never use someone else's block list, ever.
7:35 pm on Apr 9, 2014 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:May 14, 2008
posts:3091
votes: 2


I block the media bot (also from G and yandex). Seems to make little real difference apart from image indexing, which is irrelevant for my sites.
8:01 pm on Apr 9, 2014 (gmt 0)

Moderator from US 

WebmasterWorld Administrator 5+ Year Member Top Contributors Of The Month

joined:Dec 27, 2006
posts:2559
votes: 48


I have to agree with dstiles, I also block the msnbot-media bot. It is a personal preference based on their observed behavior. They never 304 for images they already have and they seem to ignore no-index header tags for images used in css, have a sweet tooth for old buttons images kept for some legacy pages - and other useless image files. I recently noticed ignoring of robots.txt so they are blocked via UA. I don't have issues with Google-Image and other allowed bots. Their sprees of requesting hundreds of non-existing files didn't help.

I also agree with keyplyr - it is best to observe the access logs for your own site and not use block lists found online. Some are quite old, full of things that never visit and busy the server just a bit more as they check visitors for everything you have in the list. Split seconds can add up in load time.
8:26 pm on Apr 9, 2014 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:5803
votes: 64


Well if you block these media bots you block your link imagery from showing up in most of their social media as well as imagery for their maps & videos in the SERP. Personally I find this quite valuable, accounting for measurable traffic.