homepage Welcome to WebmasterWorld Guest from 54.196.62.132
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
What's the difference between bing bot and Microsoft?
Microsoft IP staying too long...banned for now, what's with that?
erlandc




msg:4321841
 3:52 am on Jun 4, 2011 (gmt 0)

Blocked Microsoft IP - 131.107.151.80
Bing's ok 65.52.110.64, but why is MS sucking up my images?

from "GET /images[03/Jun/2011:17:04:25 -0700
to "GET /images[03/Jun/2011:17:04:26 -0700]

then back from:
"GET /images[03/Jun/2011:19:16:40 -0700]
to
"GET /images[03/Jun/2011:19:16:40 -0700

131.107.151.80 - - [03/Jun/2011:19:16:40 -0700] "GET /images/01_trF8F5AE.png HTTP/1.1" 200 739 "http://www.mysite.com/" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)"

Is MS the smae animal as bing bot?
Thanks
E

 

dstiles




msg:4322071
 9:00 pm on Jun 4, 2011 (gmt 0)

Block all 131.107/16. It's not an MS bot block, more of a general hosting range. Many nasties ceom from there, much as they do from Amazon AWS (see WebmasterWorld "Search Engine Spider and User Agent Identification" for details of that!).

bingbot IS msnbot - just the name changes - but I think you can control it in robots.txt by using either word. Neither word would work on the 131.107/16 bots as I doubt very much if they even look at robots.txt.

If you have a lot of scrapes/attacks on your site then I really recommend reading the "Search Engine Spider and User Agent Identification" forum hereabouts.

erlandc




msg:4322093
 10:21 pm on Jun 4, 2011 (gmt 0)

dstiles,
thanks, and blocked that ranged you mentioned.

am familiar with Amazon AWS nasties. my robots file is well maintained thanks to you & others on this great forum!

will keep your advice on Search Engine Spider and User Agent Identification section here.

thanks again!

e

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved