homepage Welcome to WebmasterWorld Guest from 54.163.84.199
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Should I block msnbot-media
shamrock




msg:4661774
 10:20 am on Apr 9, 2014 (gmt 0)

I've noticed it's blocked on this default list:
https://github.com/troubleshooter/drupal7-with-nginx/blob/master/blacklist.conf

Is this really recommended? could it hurt my indexing and ranking with bing?

 

keyplyr




msg:4661856
 5:54 pm on Apr 9, 2014 (gmt 0)

I don't block msnbot-media. It is used for several purposes including the Bing Image Search. If you don't want your images indexed, disallow your image directories in robots.txt or use an X-Robots-Tag in your directory's htaccess.

I would never use someone else's block list, ever.

dstiles




msg:4661889
 7:35 pm on Apr 9, 2014 (gmt 0)

I block the media bot (also from G and yandex). Seems to make little real difference apart from image indexing, which is irrelevant for my sites.

not2easy




msg:4661890
 8:01 pm on Apr 9, 2014 (gmt 0)

I have to agree with dstiles, I also block the msnbot-media bot. It is a personal preference based on their observed behavior. They never 304 for images they already have and they seem to ignore no-index header tags for images used in css, have a sweet tooth for old buttons images kept for some legacy pages - and other useless image files. I recently noticed ignoring of robots.txt so they are blocked via UA. I don't have issues with Google-Image and other allowed bots. Their sprees of requesting hundreds of non-existing files didn't help.

I also agree with keyplyr - it is best to observe the access logs for your own site and not use block lists found online. Some are quite old, full of things that never visit and busy the server just a bit more as they check visitors for everything you have in the list. Split seconds can add up in load time.

keyplyr




msg:4661898
 8:26 pm on Apr 9, 2014 (gmt 0)

Well if you block these media bots you block your link imagery from showing up in most of their social media as well as imagery for their maps & videos in the SERP. Personally I find this quite valuable, accounting for measurable traffic.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved