Welcome to WebmasterWorld Guest from

Forum Moderators: mack

Message Too Old, No Replies

Microsoft MSN Bot Live in the Wild!

Microsoft is Crawling



2:02 am on Jun 17, 2003 (gmt 0)

10+ Year Member

Look who's all over my sites!

"MSNBOT/0.1 (http://search.msn.com/msnbot.htm [search.msn.com])"

I've checked the IP - it's legitimate, Microsoft is crawliing the web!

IP 131.107.137.xxx


3:19 pm on Jun 24, 2003 (gmt 0)

10+ Year Member

Hi Driesie!

Sorry, but I seem to fail to see any anti-microsoft sentiment here... I read a few comments that questioned MS's ability to run a search engine of the size that they plan on their own operating systems due to lack of scalability or uptime stability, I read posts that complained that their bot seems to request pages quite often and I read of concerns what may happen if MS "owns" all searches via the already predominant interface called Internet Explorer and directly via the Windows operating systems.

You are right, Google should shape up to prepare for the competition from any side, not only MS. But will MS really bring competition to the game? Could it not be that MS is a money-hungry company that may plan the following: Get a foothold in the SE market, then use Internet Explorer and the OS to direct more searches to MSN (if possible all searches. give very cryptic instructions on how to change default searching behaviour, so that no one ever attempts to change. that'll take care of anti-trust laws...). Then tell all webhosting companies that a new add-on is available for MS Internet Information Server (IIS) that allows better indexing of only IIS sites via MSNBot.

And once about 80 percent of all searches run via MSN we'll change to paid inclusion, overnight. And the money will flow.......

Well, I might be seeing ghosts here, but I might as well see an ugly possible future, hmm?

Competition yes, but MS is no competition, it's a bully using bully tactics, in the end stifling competition! Or when were you last asked what operating system you'd like on your new computer...


The Subtle Knife

3:34 pm on Jun 24, 2003 (gmt 0)

10+ Year Member

but, will the MSN Bot index
apache based/open source web server based sites?

I think that's the quesion.


4:22 pm on Jun 24, 2003 (gmt 0)

10+ Year Member

Great, a new robots.txt will go up right now :-)

User-Agent: MSNBOT
Disallow: /

...and then I wait for MS to not obey it. I just can't imagine them getting THIS right!

I must have wrongly interpreted this an anti-microsoft feeling.

Again, don't get me wrong, I'm not a huge MS fan. I kept using NS4 until I really couldn't bare it any longer, I now use NS6 most of the time. But anyway, we're diverting.

I can see your point about the wory of MS taking over, I think my argument is that if there is better competition, they will not be able to. People keep saying that MS won the browser war because of bullying netscape, I say that they won it because NS produced a very shoddy version 4 browser and didn't move on from it for years (IE4 was very bad to, but at least got improved a lot). Point being, I'm not woried if there is good enough competition.

The other thing to keep in mind is that MS have been very succesfull at monopolysing the consumer market, but fears that they would only index IIS servers would mean they need to get the majority of webmasters behind them, which will never happen IMHO.


4:30 pm on Jun 24, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

but, will the MSN Bot index
apache based/open source web server based sites?

No problems for it with mine.



4:46 pm on Jun 24, 2003 (gmt 0)

10+ Year Member

These are stats from one of my sites for today's spiders traffic:

$ grep MSNBOT access.log -c
$ grep Scooter access.log -c
$ grep ooglebot access.log -c
$ grep slurp access.log -c
$ grep atw access.log -c

MSNBOT is on the lead (well for today, anyway) and for the first time ever the Googlebot was last, but hey, the day is not over yet ;-)

p.s. - my site is on Apache.


5:27 am on Jun 25, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

my sites haven't yet been hit by the MSNbot. Am I alone?


12:11 am on Jun 28, 2003 (gmt 0)

10+ Year Member

I only just got hit today and boy its the most agressive mainstream one I've seen for a while. It completely missed the main portion of my site and attacked an embedded forum area then went.
I'm guessing its following links (for the moment) in this case since there is a big site that links to these inner pages.



5:37 am on Jul 2, 2003 (gmt 0)

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

but, will the MSN Bot index apache based/open source web server based sites?

Yup - it's been back almost every day. Started off pretty stong taking 48 pages, but now taking only a dozen or so pages each day. Seems like it's following links from other sites, and then only those links found on that particular page.


5:56 am on Jul 2, 2003 (gmt 0)

10+ Year Member

Has anyone seen there links that were just spider in msn yet?


6:02 am on Jul 2, 2003 (gmt 0)

10+ Year Member

Hi teeceo,
I am getting some traffic from msn to very fresh sub-pages


6:42 am on Jul 2, 2003 (gmt 0)

Is the MSNBot only spidering sites that have paid for inclusion, or all sites?


6:47 am on Jul 2, 2003 (gmt 0)

10+ Year Member

Hi kascade,
I haven't paid them and they have spidered my site and given good positions to my sub-pages


3:59 pm on Jul 2, 2003 (gmt 0)

10+ Year Member

Another one of my sites got hit by MSNBOT a couple of days ago. It's already in the search results for many of its key-terms.


4:40 pm on Aug 15, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

hmm... yesterday i had both of these visiting:

1) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MSIECrawler)
2) MSNBOT/0.1 (http://search.msn.com/msnbot.htm)

The first does not seem to be a crawler at all. It had no MS IP and it requested one single file with images, css-file, js-file etc. (the latter are called by <script> tags, so no bot requests those.



1:05 pm on Aug 23, 2003 (gmt 0)

10+ Year Member

MSIECrawler is sure a strange thing. All requests for robots.txt, no actual content spidered. Last two entries:

ppp-249-217.98-62.inwind.it - - [23/Aug/2003:07:15:00 -0400] "GET /robots.txt HTTP/1.1" 200 89 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MSIECrawler)"
r79.micoks.net - - [23/Aug/2003:08:42:21 -0400] "GET /robots.txt HTTP/1.1" 200 89 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; YComp; Hotbar; .NET CLR 1.0.3705; MSIECrawler)"

And still no MSN bot visits, just the usual Slurp with few hundred daily hits trying to spider signin-signup forum forms and googlebot looking for daily updates.

This 75 message thread spans 3 pages: 75

Featured Threads

Hot Threads This Week

Hot Threads This Month