Welcome to WebmasterWorld Guest from 54.227.72.69

Forum Moderators: mack

Message Too Old, No Replies

Microsoft MSN Bot Live in the Wild!

Microsoft is Crawling

     
2:02 am on Jun 17, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:June 6, 2003
posts:67
votes: 0


Look who's all over my sites!

"MSNBOT/0.1 (http://search.msn.com/msnbot.htm [search.msn.com])"

I've checked the IP - it's legitimate, Microsoft is crawliing the web!

IP 131.107.137.xxx

3:19 pm on June 24, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:May 19, 2003
posts:70
votes: 0


Hi Driesie!

Sorry, but I seem to fail to see any anti-microsoft sentiment here... I read a few comments that questioned MS's ability to run a search engine of the size that they plan on their own operating systems due to lack of scalability or uptime stability, I read posts that complained that their bot seems to request pages quite often and I read of concerns what may happen if MS "owns" all searches via the already predominant interface called Internet Explorer and directly via the Windows operating systems.

You are right, Google should shape up to prepare for the competition from any side, not only MS. But will MS really bring competition to the game? Could it not be that MS is a money-hungry company that may plan the following: Get a foothold in the SE market, then use Internet Explorer and the OS to direct more searches to MSN (if possible all searches. give very cryptic instructions on how to change default searching behaviour, so that no one ever attempts to change. that'll take care of anti-trust laws...). Then tell all webhosting companies that a new add-on is available for MS Internet Information Server (IIS) that allows better indexing of only IIS sites via MSNBot.

And once about 80 percent of all searches run via MSN we'll change to paid inclusion, overnight. And the money will flow.......

Well, I might be seeing ghosts here, but I might as well see an ugly possible future, hmm?

Competition yes, but MS is no competition, it's a bully using bully tactics, in the end stifling competition! Or when were you last asked what operating system you'd like on your new computer...

Mozart

3:34 pm on June 24, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 20, 2003
posts:146
votes: 0


but, will the MSN Bot index
apache based/open source web server based sites?

I think that's the quesion.

4:22 pm on June 24, 2003 (gmt 0)

New User

10+ Year Member

joined:Mar 12, 2003
posts:40
votes: 0



Great, a new robots.txt will go up right now :-)

User-Agent: MSNBOT
Disallow: /

...and then I wait for MS to not obey it. I just can't imagine them getting THIS right!

I must have wrongly interpreted this an anti-microsoft feeling.

Again, don't get me wrong, I'm not a huge MS fan. I kept using NS4 until I really couldn't bare it any longer, I now use NS6 most of the time. But anyway, we're diverting.

I can see your point about the wory of MS taking over, I think my argument is that if there is better competition, they will not be able to. People keep saying that MS won the browser war because of bullying netscape, I say that they won it because NS produced a very shoddy version 4 browser and didn't move on from it for years (IE4 was very bad to, but at least got improved a lot). Point being, I'm not woried if there is good enough competition.

The other thing to keep in mind is that MS have been very succesfull at monopolysing the consumer market, but fears that they would only index IIS servers would mean they need to get the majority of webmasters behind them, which will never happen IMHO.

4:30 pm on June 24, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 21, 2003
posts:2355
votes: 0


but, will the MSN Bot index
apache based/open source web server based sites?

No problems for it with mine.

Regards,
Brent

4:46 pm on June 24, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Sept 11, 2002
posts:42
votes: 0


These are stats from one of my sites for today's spiders traffic:

$ grep MSNBOT access.log -c
18
$ grep Scooter access.log -c
16
$ grep ooglebot access.log -c
1
$ grep slurp access.log -c
7
$ grep atw access.log -c
2

MSNBOT is on the lead (well for today, anyway) and for the first time ever the Googlebot was last, but hey, the day is not over yet ;-)

p.s. - my site is on Apache.

5:27 am on June 25, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 3, 2003
posts:961
votes: 0


my sites haven't yet been hit by the MSNbot. Am I alone?
12:11 am on June 28, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 15, 2003
posts:169
votes: 0


I only just got hit today and boy its the most agressive mainstream one I've seen for a while. It completely missed the main portion of my site and attacked an embedded forum area then went.
I'm guessing its following links (for the moment) in this case since there is a big site that links to these inner pages.

Rob

5:37 am on July 2, 2003 (gmt 0)

Moderator from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:6661
votes: 128


but, will the MSN Bot index apache based/open source web server based sites?

Yup - it's been back almost every day. Started off pretty stong taking 48 pages, but now taking only a dozen or so pages each day. Seems like it's following links from other sites, and then only those links found on that particular page.

5:56 am on July 2, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:June 15, 2002
posts:396
votes: 0


Has anyone seen there links that were just spider in msn yet?
6:02 am on July 2, 2003 (gmt 0)

Full Member

10+ Year Member

joined:June 15, 2003
posts:302
votes: 0


Hi teeceo,
I am getting some traffic from msn to very fresh sub-pages

kascade

6:42 am on July 2, 2003 (gmt 0)

Inactive Member
Account Expired

 
 


Is the MSNBot only spidering sites that have paid for inclusion, or all sites?
6:47 am on July 2, 2003 (gmt 0)

Full Member

10+ Year Member

joined:June 15, 2003
posts:302
votes: 0


Hi kascade,
I haven't paid them and they have spidered my site and given good positions to my sub-pages
3:59 pm on July 2, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Sept 11, 2002
posts:42
votes: 0


Another one of my sites got hit by MSNBOT a couple of days ago. It's already in the search results for many of its key-terms.
4:40 pm on Aug 15, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 15, 2003
posts:2408
votes: 5


hmm... yesterday i had both of these visiting:

1) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MSIECrawler)
2) MSNBOT/0.1 (http://search.msn.com/msnbot.htm)

The first does not seem to be a crawler at all. It had no MS IP and it requested one single file with images, css-file, js-file etc. (the latter are called by <script> tags, so no bot requests those.

/claus

1:05 pm on Aug 23, 2003 (gmt 0)

New User

10+ Year Member

joined:Aug 21, 2003
posts:13
votes: 0


MSIECrawler is sure a strange thing. All requests for robots.txt, no actual content spidered. Last two entries:

ppp-249-217.98-62.inwind.it - - [23/Aug/2003:07:15:00 -0400] "GET /robots.txt HTTP/1.1" 200 89 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MSIECrawler)"
r79.micoks.net - - [23/Aug/2003:08:42:21 -0400] "GET /robots.txt HTTP/1.1" 200 89 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; YComp 5.0.0.0; Hotbar 4.3.2.0; .NET CLR 1.0.3705; MSIECrawler)"

And still no MSN bot visits, just the usual Slurp with few hundred daily hits trying to spider signin-signup forum forms and googlebot looking for daily updates.

This 75 message thread spans 3 pages: 75
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members