Forum Moderators: mack

Message Too Old, No Replies

MSN Bot Smarter than Googlebot

when deep crawling

         

dvduval

3:00 am on Jan 23, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I continue to see MSN bot doing a better job at finding the best deep links. Let me give you a great example:
I have this new dictionary about widgets. It is still a new dictionary so some of the letters (each letter is a page) do not have definitions yet.
* MSN Bot crawled only the letters that contained definitions (about 14)
* Google Bot crawled 13 pages total, and only 3 contained definitions. Google crawled Q and X but not A or M. Most of the pages were "duplicate content" because they were blank with no definitions.

This is not the only case I have seen like this, but it is a prime example.

eyezshine

5:21 am on Jan 23, 2005 (gmt 0)

10+ Year Member



I wish google would index deeper into my site where all the good pages are but it don't go very deep. Maybe 3-4 levels deep and that's it.

I think the higher your PR, the deeper it will index but I sure can't tell because it never goes very deep.

MSN bot spiders way deeper but doesn't index the deeper pages either.

to_tall

11:37 pm on Jan 26, 2005 (gmt 0)

10+ Year Member



>"I continue to see MSN bot doing a better job at finding the best deep links"

This is a joke, right? I can't get them to even include my index page in their serps even though the bot has found it weeks ago and is indexed as often as two times a day--and it was listed high for a while in the serps, now it's gone. The only ones I see doing a decent job these days are Yahoo and Google.I never though I would say a kind thing about Google but it is clear they know what they are doing when it comes to indexing.

RichTC

12:04 am on Jan 27, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



to-tall

now your having a laugh with us?

MSNs new technology bot is far, far better at indexing. Googlebot on the other hand takes information and then does nothing with it in the SERPS.

Also, MSN bot is going deeper into sites, FACT

to_tall

12:17 am on Jan 27, 2005 (gmt 0)

10+ Year Member



>"now your having a laugh with us?

MSNs new technology bot is far, far better at indexing. Googlebot on the other hand takes information and then does nothing with it in the SERPS.
Also, MSN bot is going deeper into sites, FACT "

Just because you say it is a fact does not make it so. If that were the case, they would have more results for my sites: Site:search, and they don't, not even close. Google has three times as many, and it's not like I even like google, just giving the true facts, Msn is the worst of the big three SEs and is getting worse every time I look at it.
I think they use a dart board to pick ranking, it makes mose sense then most of the stuff I have read on this board, That is one SE that sould have stayed off-line until they get it fixed.

balam

12:31 am on Jan 27, 2005 (gmt 0)

10+ Year Member



> I can't get them to even include my index page in their serps even though the bot has found it weeks ago and is indexed as often as two times a day--and it was listed high for a while in the serps, now it's gone.

> If that were the case, they would have more results for my sites [...]

Ah yes, sounds familiar - the old "Search Engine X stinks because I'm not top of the SERPs and my whole site isn't in the index" complaint.

And of course it couldn't have anything to do with the complainee's site, it's always the search engine's fault.

to_tall

12:37 am on Jan 27, 2005 (gmt 0)

10+ Year Member



>"Ah yes, sounds familiar - the old "Search Engine X stinks because I'm not top of the SERPs and my whole site isn't in the index" complaint."

My site is doing better in Msn then Google, so what is your opinion now?

to_tall

12:44 am on Jan 27, 2005 (gmt 0)

10+ Year Member



Even the worst SE there is has the smarts to include the index page. On wait, that is the point here, the worst SE doesn't have the sense to include the index page, ha.

balam

1:30 am on Jan 27, 2005 (gmt 0)

10+ Year Member



> so what is your opinion now?

Before I offer it, please correct me if I'm wrong...

  • MSN has indexed one-third of the pages that Google has indexed.
  • MSN has crawled your homepage, it ranked well, but now you can't find it.
  • Google has crawled and indexed your homepage, and you can find it.
  • Both MSN & Google are actively crawling your site.
  • You rank well in MSN for the pages they have indexed.
  • You rank poorly in Google for the pages they have indexed.
  • Your site is weeks old. (How many?)
  • You use no black or gray-hat SEO techniques.
  • Since MSN doesn't return your homepage in the SERPs, it's the "worst SE," and that is the basis of your complaint.
  • Since MSN doesn't return your homepage, it can't be deep-crawling others' sites, despite the evidence.

You have how many websites?

to_tall

5:26 am on Jan 27, 2005 (gmt 0)

10+ Year Member



>"Before I offer it, please correct me if I'm wrong... "
>"You rank well in MSN for the pages they have indexed."

NO I DON'T RANK WELL, JUST BETTER THEN I DO IN GOOGLE. BUT NOT NEARLY AS WELL AS I DO IN A SE OF TRUE QUALITY--- (YAHOO), THEY ARE GOOD BECAUSE THEY BASE THEIR RANKING ON RELEVANCY, THAT IS WHAT SHOULD MATTER, NOT HOW MANY LINK EXCHANGES YOU ARRANGE, OR PRODUCE FOR YOURSELF, WHICH I THINK HAPPENS A LOT.

>"You rank poorly in Google for the pages they have indexed. "

BECAUSE I DON'T SOLICIT LINKS, OR RUN A LINKS FARM,

>Your site is weeks old. (How many?)

16 MONTHS AND WHY SHOULD THIS MATTER?

>You use no black or gray-hat SEO techniques.

NO I DON'T, AND THIS INCLUDES TRADING LINKS AND LOADING UP THE NUMBER OF PAGES TO INCREASE LIKELIHOOD OF GETTING LISTED--THE LAST TWO ARE, IN MHO, WHAT IS WRONG WITH THE WEB IN GENERAL, AND GOOGLE IN PARTICULAR

>"Since MSN doesn't return your homepage in the SERPs, it's the "worst SE," and that is the basis of your complaint.

NO, THEY ARE THE WORST BECAUSE THEY SIMPLY ARE, YOU NEED ONLY LOOK BEYOND YOUR SUCCESS WITH THEM TO SEE THIS, THE DART BOARD THEORY IS WHAT I THINK BEST DESCRIBES THE WAY IN WHICH THEY RANK, IT HAS NOTHING TO DO WITH THE SITE'S REVERENCE OR THE OVERALL QUALITY OF THE SITE

>"Since MSN doesn't return your homepage, it can't be deep-crawling others' sites, despite the evidence."

NOT SHOWING AN INDEX PAGE IS JUST PLAIN STUPID, IF THEY CAN'T DEEP CRAWL HOW CAN THEY BE GETTING GOOD RESULTS? WHO'S EVIDENCE, FROM WHAT I READ, THEY ARE NOT DEEP-CRAWLING

>"You have how many websites? "

THIS IS OF NO RELEVANCE, I BASE MY OPINION ON AN OVERVIEW OF THE RESULTS FOR ALL SITE SEARCHES, UNLIKE YOU, WHO MUST BE (FOR NOW) RANKING WELL IN THEIR SERPS.

eyezshine

9:44 am on Jan 27, 2005 (gmt 0)

10+ Year Member



I read in a Blog on MSN that they don't index pages deep in your site unless there are other websites linking to your internal pages.

From what I can tell, they go by your link popularity which tells them how deep to go into your site.

If other pages link to your internal pages it will index those pages. Otherwise it will just index so deep according to the amount of links pointing to your home page.

They are basically trying to be like google and calculate the odds of a surfer stumbling onto your page. The better the odds (More links pointing to a page) the higher that single page will rank.

If going to your home page is the only way to get to your internal page that is 6 levels deep into your site (Takes 6 clicks to get to it) then MSN bot will spider it but not index it.

I was excited at first when I seen MSNBot spidering so deep in my sites but now I see what's really going on.

Their suggestion was to create a site map. Go figure?

Macro

9:53 am on Jan 27, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



to_tall, I turned off my attention as soon as I saw all the capitals. When you resort to shouting to make your point ... you've lost the argument (and the plot).

Liane

10:05 am on Jan 27, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I wish google would index deeper into my site where all the good pages are but it don't go very deep. Maybe 3-4 levels deep and that's it.

Do you have a site map? If not, I suggest you build one. This helps the bots to find all your pages regardless of how deep the links are buried.

eyezshine

10:26 am on Jan 27, 2005 (gmt 0)

10+ Year Member



I don't have a server big enought to hold the site map I would need to list all my pages.

I found a better way and in theory it should work even better than a site map.

I have 5 different sites that are structured exactly the same with different content. But I can link my internal pages of the other sites to the internal pages of this site.

I figure I can link to 2 other sites pages from one page and do it without cross linking the internal pages which will cause 2 links from other websites pages to link to every page on the one site.

It's my "external internal linking theory"!

In plain english...

All of the internal pages of site A would link to all of the internal pages of sites B and C.

B would link to C and D
C would link to D and E
D would link to E and F
F would link to F and A
A would link to B and C
etc...

This would create 2 external pages linking to all of my internal pages of all my 5 sites.

Does it make any sense?

PCInk

11:06 am on Jan 27, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Actually, I agree with to_tall from my experience:

I can't get them to even include my index page in their serps even though the bot has found it weeks ago and is indexed as often as two times a day

I'll explain what has happened to my site and it may make more sense:

1) MSN crawls the pages repeatedly but I can never find these pages in the search by entering a search term
2) The pages CAN be found only by searching on the domain name
3) Weeks later (months from when the beta was probably first released to the public) some pages can now be searched upon.

It appears that they index the pages separately from indexing the content of the pages or from building a dictionary list of terms on your page.

The fact that it is crawling does appear that your site will be listed, eventually.

brixton

11:38 am on Jan 27, 2005 (gmt 0)



why you all moaning the guys from MS say they are on beta stage so the results you see are all in beta.When they will get to gama and then to alpha ,delta, kapa and so on we will see what will hapen.

brixton

11:41 am on Jan 27, 2005 (gmt 0)



to continue my last post with A,B,C the new Ms search looks to me like a baby in the kindergarden that learns the first ABC steps.

Bentler

2:42 pm on Jan 27, 2005 (gmt 0)

10+ Year Member



I just searched MSN on an obscure keyphrase and within minutes MSNbot spidered one of my pages containing the phrase. Just that one page, which wasn't listed in MSN SERPs. Coincidence?

balam

3:12 pm on Jan 27, 2005 (gmt 0)

10+ Year Member



> I turned off my attention

I was suddenly deafened.

I said, I WAS SUDDENLY DEAFENED!