homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Microsoft / Bing Search Engine News
Forum Library, Charter, Moderators: mack

Bing Search Engine News Forum

This 75 message thread spans 3 pages: < < 75 ( 1 2 [3]     
Microsoft MSN Bot Live in the Wild!
Microsoft is Crawling

 2:02 am on Jun 17, 2003 (gmt 0)

Look who's all over my sites!

"MSNBOT/0.1 (http://search.msn.com/msnbot.htm [search.msn.com])"

I've checked the IP - it's legitimate, Microsoft is crawliing the web!

IP 131.107.137.xxx



 3:19 pm on Jun 24, 2003 (gmt 0)

Hi Driesie!

Sorry, but I seem to fail to see any anti-microsoft sentiment here... I read a few comments that questioned MS's ability to run a search engine of the size that they plan on their own operating systems due to lack of scalability or uptime stability, I read posts that complained that their bot seems to request pages quite often and I read of concerns what may happen if MS "owns" all searches via the already predominant interface called Internet Explorer and directly via the Windows operating systems.

You are right, Google should shape up to prepare for the competition from any side, not only MS. But will MS really bring competition to the game? Could it not be that MS is a money-hungry company that may plan the following: Get a foothold in the SE market, then use Internet Explorer and the OS to direct more searches to MSN (if possible all searches. give very cryptic instructions on how to change default searching behaviour, so that no one ever attempts to change. that'll take care of anti-trust laws...). Then tell all webhosting companies that a new add-on is available for MS Internet Information Server (IIS) that allows better indexing of only IIS sites via MSNBot.

And once about 80 percent of all searches run via MSN we'll change to paid inclusion, overnight. And the money will flow.......

Well, I might be seeing ghosts here, but I might as well see an ugly possible future, hmm?

Competition yes, but MS is no competition, it's a bully using bully tactics, in the end stifling competition! Or when were you last asked what operating system you'd like on your new computer...


The Subtle Knife

 3:34 pm on Jun 24, 2003 (gmt 0)

but, will the MSN Bot index
apache based/open source web server based sites?

I think that's the quesion.


 4:22 pm on Jun 24, 2003 (gmt 0)

Great, a new robots.txt will go up right now :-)

User-Agent: MSNBOT
Disallow: /

...and then I wait for MS to not obey it. I just can't imagine them getting THIS right!

I must have wrongly interpreted this an anti-microsoft feeling.

Again, don't get me wrong, I'm not a huge MS fan. I kept using NS4 until I really couldn't bare it any longer, I now use NS6 most of the time. But anyway, we're diverting.

I can see your point about the wory of MS taking over, I think my argument is that if there is better competition, they will not be able to. People keep saying that MS won the browser war because of bullying netscape, I say that they won it because NS produced a very shoddy version 4 browser and didn't move on from it for years (IE4 was very bad to, but at least got improved a lot). Point being, I'm not woried if there is good enough competition.

The other thing to keep in mind is that MS have been very succesfull at monopolysing the consumer market, but fears that they would only index IIS servers would mean they need to get the majority of webmasters behind them, which will never happen IMHO.


 4:30 pm on Jun 24, 2003 (gmt 0)

but, will the MSN Bot index
apache based/open source web server based sites?

No problems for it with mine.



 4:46 pm on Jun 24, 2003 (gmt 0)

These are stats from one of my sites for today's spiders traffic:

$ grep MSNBOT access.log -c
$ grep Scooter access.log -c
$ grep ooglebot access.log -c
$ grep slurp access.log -c
$ grep atw access.log -c

MSNBOT is on the lead (well for today, anyway) and for the first time ever the Googlebot was last, but hey, the day is not over yet ;-)

p.s. - my site is on Apache.


 5:27 am on Jun 25, 2003 (gmt 0)

my sites haven't yet been hit by the MSNbot. Am I alone?


 12:11 am on Jun 28, 2003 (gmt 0)

I only just got hit today and boy its the most agressive mainstream one I've seen for a while. It completely missed the main portion of my site and attacked an embedded forum area then went.
I'm guessing its following links (for the moment) in this case since there is a big site that links to these inner pages.



 5:37 am on Jul 2, 2003 (gmt 0)

but, will the MSN Bot index apache based/open source web server based sites?

Yup - it's been back almost every day. Started off pretty stong taking 48 pages, but now taking only a dozen or so pages each day. Seems like it's following links from other sites, and then only those links found on that particular page.


 5:56 am on Jul 2, 2003 (gmt 0)

Has anyone seen there links that were just spider in msn yet?


 6:02 am on Jul 2, 2003 (gmt 0)

Hi teeceo,
I am getting some traffic from msn to very fresh sub-pages


 6:42 am on Jul 2, 2003 (gmt 0)

Is the MSNBot only spidering sites that have paid for inclusion, or all sites?


 6:47 am on Jul 2, 2003 (gmt 0)

Hi kascade,
I haven't paid them and they have spidered my site and given good positions to my sub-pages


 3:59 pm on Jul 2, 2003 (gmt 0)

Another one of my sites got hit by MSNBOT a couple of days ago. It's already in the search results for many of its key-terms.


 4:40 pm on Aug 15, 2003 (gmt 0)

hmm... yesterday i had both of these visiting:

1) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MSIECrawler)
2) MSNBOT/0.1 (http://search.msn.com/msnbot.htm)

The first does not seem to be a crawler at all. It had no MS IP and it requested one single file with images, css-file, js-file etc. (the latter are called by <script> tags, so no bot requests those.



 1:05 pm on Aug 23, 2003 (gmt 0)

MSIECrawler is sure a strange thing. All requests for robots.txt, no actual content spidered. Last two entries:

ppp-249-217.98-62.inwind.it - - [23/Aug/2003:07:15:00 -0400] "GET /robots.txt HTTP/1.1" 200 89 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MSIECrawler)"
r79.micoks.net - - [23/Aug/2003:08:42:21 -0400] "GET /robots.txt HTTP/1.1" 200 89 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; YComp; Hotbar; .NET CLR 1.0.3705; MSIECrawler)"

And still no MSN bot visits, just the usual Slurp with few hundred daily hits trying to spider signin-signup forum forms and googlebot looking for daily updates.

This 75 message thread spans 3 pages: < < 75 ( 1 2 [3]
Global Options:
 top home search open messages active posts  

Home / Forums Index / Microsoft / Bing Search Engine News
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved