homepage Welcome to WebmasterWorld Guest from 54.145.209.77
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Should I block msnbot-media
shamrock

5+ Year Member



 
Msg#: 4661772 posted 10:20 am on Apr 9, 2014 (gmt 0)

I've noticed it's blocked on this default list:
https://github.com/troubleshooter/drupal7-with-nginx/blob/master/blacklist.conf

Is this really recommended? could it hurt my indexing and ranking with bing?

 

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4661772 posted 5:54 pm on Apr 9, 2014 (gmt 0)

I don't block msnbot-media. It is used for several purposes including the Bing Image Search. If you don't want your images indexed, disallow your image directories in robots.txt or use an X-Robots-Tag in your directory's htaccess.

I would never use someone else's block list, ever.

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4661772 posted 7:35 pm on Apr 9, 2014 (gmt 0)

I block the media bot (also from G and yandex). Seems to make little real difference apart from image indexing, which is irrelevant for my sites.

not2easy

WebmasterWorld Administrator 5+ Year Member Top Contributors Of The Month



 
Msg#: 4661772 posted 8:01 pm on Apr 9, 2014 (gmt 0)

I have to agree with dstiles, I also block the msnbot-media bot. It is a personal preference based on their observed behavior. They never 304 for images they already have and they seem to ignore no-index header tags for images used in css, have a sweet tooth for old buttons images kept for some legacy pages - and other useless image files. I recently noticed ignoring of robots.txt so they are blocked via UA. I don't have issues with Google-Image and other allowed bots. Their sprees of requesting hundreds of non-existing files didn't help.

I also agree with keyplyr - it is best to observe the access logs for your own site and not use block lists found online. Some are quite old, full of things that never visit and busy the server just a bit more as they check visitors for everything you have in the list. Split seconds can add up in load time.

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4661772 posted 8:26 pm on Apr 9, 2014 (gmt 0)

Well if you block these media bots you block your link imagery from showing up in most of their social media as well as imagery for their maps & videos in the SERP. Personally I find this quite valuable, accounting for measurable traffic.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved