Welcome to WebmasterWorld Guest from 220.127.116.11
Forum Moderators: mack
Sorry, but I seem to fail to see any anti-microsoft sentiment here... I read a few comments that questioned MS's ability to run a search engine of the size that they plan on their own operating systems due to lack of scalability or uptime stability, I read posts that complained that their bot seems to request pages quite often and I read of concerns what may happen if MS "owns" all searches via the already predominant interface called Internet Explorer and directly via the Windows operating systems.
You are right, Google should shape up to prepare for the competition from any side, not only MS. But will MS really bring competition to the game? Could it not be that MS is a money-hungry company that may plan the following: Get a foothold in the SE market, then use Internet Explorer and the OS to direct more searches to MSN (if possible all searches. give very cryptic instructions on how to change default searching behaviour, so that no one ever attempts to change. that'll take care of anti-trust laws...). Then tell all webhosting companies that a new add-on is available for MS Internet Information Server (IIS) that allows better indexing of only IIS sites via MSNBot.
And once about 80 percent of all searches run via MSN we'll change to paid inclusion, overnight. And the money will flow.......
Well, I might be seeing ghosts here, but I might as well see an ugly possible future, hmm?
Competition yes, but MS is no competition, it's a bully using bully tactics, in the end stifling competition! Or when were you last asked what operating system you'd like on your new computer...
Great, a new robots.txt will go up right now :-)
...and then I wait for MS to not obey it. I just can't imagine them getting THIS right!
I must have wrongly interpreted this an anti-microsoft feeling.
Again, don't get me wrong, I'm not a huge MS fan. I kept using NS4 until I really couldn't bare it any longer, I now use NS6 most of the time. But anyway, we're diverting.
I can see your point about the wory of MS taking over, I think my argument is that if there is better competition, they will not be able to. People keep saying that MS won the browser war because of bullying netscape, I say that they won it because NS produced a very shoddy version 4 browser and didn't move on from it for years (IE4 was very bad to, but at least got improved a lot). Point being, I'm not woried if there is good enough competition.
The other thing to keep in mind is that MS have been very succesfull at monopolysing the consumer market, but fears that they would only index IIS servers would mean they need to get the majority of webmasters behind them, which will never happen IMHO.
$ grep MSNBOT access.log -c
$ grep Scooter access.log -c
$ grep ooglebot access.log -c
$ grep slurp access.log -c
$ grep atw access.log -c
MSNBOT is on the lead (well for today, anyway) and for the first time ever the Googlebot was last, but hey, the day is not over yet ;-)
p.s. - my site is on Apache.
joined:Sept 26, 2001
but, will the MSN Bot index apache based/open source web server based sites?
Yup - it's been back almost every day. Started off pretty stong taking 48 pages, but now taking only a dozen or so pages each day. Seems like it's following links from other sites, and then only those links found on that particular page.
1) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MSIECrawler)
2) MSNBOT/0.1 (http://search.msn.com/msnbot.htm)
The first does not seem to be a crawler at all. It had no MS IP and it requested one single file with images, css-file, js-file etc. (the latter are called by <script> tags, so no bot requests those.
ppp-249-217.98-62.inwind.it - - [23/Aug/2003:07:15:00 -0400] "GET /robots.txt HTTP/1.1" 200 89 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MSIECrawler)"
r79.micoks.net - - [23/Aug/2003:08:42:21 -0400] "GET /robots.txt HTTP/1.1" 200 89 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; YComp 18.104.22.168; Hotbar 22.214.171.124; .NET CLR 1.0.3705; MSIECrawler)"
And still no MSN bot visits, just the usual Slurp with few hundred daily hits trying to spider signin-signup forum forms and googlebot looking for daily updates.