homepage Welcome to WebmasterWorld Guest from 54.167.179.48
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
What's the difference between bing bot and Microsoft?
Microsoft IP staying too long...banned for now, what's with that?
erlandc

10+ Year Member



 
Msg#: 4321839 posted 3:52 am on Jun 4, 2011 (gmt 0)

Blocked Microsoft IP - 131.107.151.80
Bing's ok 65.52.110.64, but why is MS sucking up my images?

from "GET /images[03/Jun/2011:17:04:25 -0700
to "GET /images[03/Jun/2011:17:04:26 -0700]

then back from:
"GET /images[03/Jun/2011:19:16:40 -0700]
to
"GET /images[03/Jun/2011:19:16:40 -0700

131.107.151.80 - - [03/Jun/2011:19:16:40 -0700] "GET /images/01_trF8F5AE.png HTTP/1.1" 200 739 "http://www.mysite.com/" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)"

Is MS the smae animal as bing bot?
Thanks
E

 

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4321839 posted 9:00 pm on Jun 4, 2011 (gmt 0)

Block all 131.107/16. It's not an MS bot block, more of a general hosting range. Many nasties ceom from there, much as they do from Amazon AWS (see WebmasterWorld "Search Engine Spider and User Agent Identification" for details of that!).

bingbot IS msnbot - just the name changes - but I think you can control it in robots.txt by using either word. Neither word would work on the 131.107/16 bots as I doubt very much if they even look at robots.txt.

If you have a lot of scrapes/attacks on your site then I really recommend reading the "Search Engine Spider and User Agent Identification" forum hereabouts.

erlandc

10+ Year Member



 
Msg#: 4321839 posted 10:21 pm on Jun 4, 2011 (gmt 0)

dstiles,
thanks, and blocked that ranged you mentioned.

am familiar with Amazon AWS nasties. my robots file is well maintained thanks to you & others on this great forum!

will keep your advice on Search Engine Spider and User Agent Identification section here.

thanks again!

e

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved