Forum Moderators: mack
We touched on this during some other threads and i understand from msndudes post that they are trying to do something about it but currently the bot just skims the top of the larger websites.
Im finding that on the large sites the bot takes about 10-15% if your lucky of the overall content.
Also, becasue i think their spam filters dont work correctly, some good large sites can have say 4,000 pages in the index one day and zap 800 the next! In some cases a large page count can almost vanish on a quality site over a few days and so the cycle repeats!
The very large sites 100,000 pages plus dont stand a chance with msn it simply cant cope.
Here is an example of bot activity on a large site we have worked on that is quality, clean and rich in content, it shows perfectly how far away msn are currently with their search bot technology:-
Period 1-20th Feb
Googlebot 1,010,724 Hits
Inktomi Slurp 384,792 Hits
wisenutbot 34,035 Hits
Gigabot 18,221 Hits
Askjeeves 16,432 Hits
psbot 11,221 Hits
msnbot 9,024 Hits
alexa 5,884 Hits
udmsearch 3,889 Hits
voyger 1,220 Hits
Nomad 884 Hits
Others 12,889
We see a similar pattern on other sites we work on
In this example, even gigabot is twice as active and is able to index far more pages than msn bot, likewise even wisenut is working three times as hard.
Obviously out at the front is googlebot which is a massive 100 times more effective, then its Yahoo, you will recall yahoo recently saying that they were not able to get close to google and right they are, as for msn i think bill gates is in cloud Cookoo land if he thinks msn search is going to compete in the short term - unless they improve the bot technology they will never get there imo.
I think if they sort out the search bot deep indexing technology they may have half a chance but its a poor show currently when the likes of gigabot and wisenut can out perform them.
One last thing, msn need to be aware that to be the market leader their bot needs to be better and more effective than googlebot, after all it has a lot of years catching up to do and unless it can perform better than googlebot it can NEVER catch up with the data indexed by google.
However search engines engine robots spider sites depending on its 'importance' according to their algorithm.
as an example over a few sites, i see yahoo slurp hammering BUT not indexing pages (mess...pages age months old, only 10 to 15% indexed) then MSN at the second position with about 80% pages indexed then Google is the one who visits less but has 100% of pages indexed...don't ask me, just observations.
i think its more like small website owners crow over having a 100 page site and msn indexing all of it in a week.
A much greater number of posters at webmasterworld are involved in small sites than large sites imo
If you take say a handfull of top high content sites and use the site command on msn you will quickly notice that it isnt able to cashe the majority of a large sites pages, its that simple.
On the other hand if you knock together say a 50 page site msn will be the first to cashe every page and have it in the index.
With both Google and yahoo it will be in the sand for a given period of time.
