homepage Welcome to WebmasterWorld Guest from 54.167.179.48
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Microsoft / Bing Search Engine News
Forum Library, Charter, Moderators: mack

Bing Search Engine News Forum

    
Is this an MSN Search bot?
Strange behaviour
geekay

10+ Year Member



 
Msg#: 241 posted 5:38 pm on Nov 16, 2004 (gmt 0)

One page on my site was requested by an IP belonging to Microsoft Corporation. The User Agent is very ordinary looking.

On the next row in my log an IP of XO Communications requests the same page. (The time stamp is actually 3 sec. earlier, but I don't know if the time in the log is the time of full delivery or of request - it's a big 250K file.) This time the UA is special: "Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; ...../1.0 )".

Then 245 more files (html, jpeg, all different) are requested in 30 minutes (when interrupted) with the very same UA, but the IP's are either one belonging to XO, Sprintlink/Microsoft or AT&T. Three different IP's are used and only one or two requests are made with the same IP before it's changed. Alternating between the three IP's goes on for the whole session. My robots.txt is not requested.

This doesn't look like a DoS. Neither Site Grabbing (no need to use three IP's for my tiny site), and if IP's are spoofed grabbing wouldn't work. But why wouldn't an MSN Search bot identify itself properly? The clues point at MS, but they can't have any professional interest in my site.

 

bull

10+ Year Member



 
Msg#: 241 posted 7:30 pm on Nov 17, 2004 (gmt 0)

[webmasterworld.com...]
[webmasterworld.com...]

Bad one. I denied the entire MS IP range for any User-agent that does not begin with msnbot.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Microsoft / Bing Search Engine News
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved