Welcome to WebmasterWorld Guest from 54.205.75.60

Forum Moderators: mack

Message Too Old, No Replies

MSN now very slow to include all pages

It used to index them all within weeks

   
12:13 pm on Jan 11, 2007 (gmt 0)

10+ Year Member



I have submitted lots of websites over several years to MSN.co.uk and it has always crawled and indexed all of my web pages very quickly ie within weeks.

Since September 2006 or the end of last year from a UK perspective MSN has started to index very slowly and I am seeing 1 page indexed so far for each of my new sites.

I am hoping that in its next cycle (in another month?) MSN will index all the other pages linked to the page I submitted but fear it will just keeping showing 1 page for each site?

Anybody else seeing dramticvally slower inclusion of pages compared to 6 months ago?

3:14 pm on Jan 12, 2007 (gmt 0)

5+ Year Member



You Bet...
It's taking months compared to weeks last year.
Perhaps the new MSNDude can address this, why don't you post a new thread to attention of new MSNDude.

Fish Texas

12:19 am on Jan 13, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I'm seeing the exact same thing. At one point last year I had to post a question in the robots.txt section of this forum for help in SLOWING DOWN the msn bot. Now, I feel like I need to send out a special invitation with flowers & chocolate. Whatever is happening is frustrating and I can only hope it's because they are planning to release a better operating search service in the near future. Time will (presumably) tell.

...............................

12:22 pm on Jan 15, 2007 (gmt 0)

10+ Year Member



It seems that submitting every single page one at a time using the MSN submission page works at getting more pages listed fairly quickly (within weeks)

However surely submitting one page (home page) should result in a crawl of all pages linked to from the home page? (as it used to last year)

I have submitted single pages several months ago for several sites and still only one pages is showing per site. It is hard work having to manually submit each and every page!

MSNDude please fix your very very slow crawling or worse still that MSN bot does not even follow same site links anymore!

7:14 pm on Jan 15, 2007 (gmt 0)

5+ Year Member



ive noticed this with MSN actually over the last two months. It bizarrely indexes some of our less important pages and hasnt touched any of the user generated good stuff. Google seems to index that with no problem.

Would be interesting to know what is happening.

3:46 am on Jan 16, 2007 (gmt 0)

5+ Year Member



Maybe MSN and Live are transfering searching functions and cause some bots connection problems.

Sites de-indexed problems appeared more and more recently.

You may try this url to submit sites :: search.live.com/docs/submit.aspx

I think they want you to submit to LIVE.

[edited by: Ezer at 3:47 am (utc) on Jan. 16, 2007]

4:27 am on Jan 16, 2007 (gmt 0)

10+ Year Member



Just so this is clear -- we only have one search engine and it's Live Search.

Live Search simply refers to the search engine as it has existed since the release in September, prior to which it was referred to as MSN Search.

Our one bot is MSNBot, and I'm not aware of anything new that might result in slow indexing. There's always some variance in terms of time for this.

11:47 am on Jan 16, 2007 (gmt 0)

10+ Year Member



"I'm not aware of anything new that might result in slow indexing"

Dude, my site is two years old. Yahoo has 120,938 pages indexed, MSN has 94! No problem?

2:53 pm on Jan 17, 2007 (gmt 0)

10+ Year Member



MSNDude your bot is broken without a doubt I have written hundreds of websites over many years and all pages get indexed within weeks until the last few months.....

Only the page I manually submit gets indexed and no further crawl or indexing of the other pages linked to from that page are indexed even after several months.

I can get them all included if I manually submit one at a time but surely your bot should crawl links from a homepage and include them as it used to last year.

This cannot be due to an algo change as I can get the pages in if submit each one individually.

I am submitting here [search.msn.co.uk...]

Thanks for reading this thread and taking the trouble to reply.

3:04 pm on Jan 17, 2007 (gmt 0)

10+ Year Member



PS Google and Yahoo are indexing these sites and all pages no problem.
7:24 pm on Jan 18, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



There is no question in my mind that something about the behavior of the msnbot has changed over the past several months. Sites of ours that it had fully indexed are now gone, other sites are included but are only partially indexed. So something has happened, as the observations in this thread confirm.

ps. I have had limited success in getting pages back into MSN by submitting them one at a time, but have also found that doing that slow & tedious process is still NOT restoring (so far at least) many of the pages that used to be included.

Also, I hasten to add that NO significant changes had been made to any of those pages that are now missing -- one day they were in the index, the next day gone.

...........................

11:53 pm on Jan 18, 2007 (gmt 0)

10+ Year Member



The reason is probably that MSN got complaints from webmasters about the MSNbot being too greedy. So now it's very tentative with its spidering.

Read it on a site while I was researching robots.txt tutorials.

And no, I can't remember the site name.

1:32 am on Jan 19, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



The reason is probably that MSN got complaints from webmasters about the MSNbot being too greedy. So now it's very tentative with its spidering.

That may explain some of what is going on. Still, it is curious that once well-indexed sites are dropping out entirely, or, the number of pages is reduced from close to 100% down to 5-10%. I still have big hopes that "Live" will eventually become more of a competitor to Google, so the sooner that they get back on track the better it will be for them and for us.

.....................................

1:59 am on Jan 19, 2007 (gmt 0)

10+ Year Member



They are also banning sites. This happened to me. Search on threads I've posted in recently for the possible causes.
3:17 am on Jan 19, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



They are also banning sites

If what you say is true then that would mean that instead of trying to develop a search engine that brings back the highest quality results based on an algo that is truly superior to Google, they would be intending to follow Google's example and start punishing sites OUT of their database, which of course will mean even less of an index from which to find the best possible results. I really hope that is not the case -- the web needs a viable alternative to Google, not a pale imitation.

...............................

12:41 pm on Jan 19, 2007 (gmt 0)

10+ Year Member



I disagree.

Put yourself in a search engine's shoes. Spammers put millions of dud pages on the SE's hard disks. You can define quality by certain criteria, and spam by other.

You'd be happy to set a filter that deletes most spam, with some collateral damage. So some webmasters whine, so what? If your SE is tanking, like MSN and Yahoo, that's not a priority.

I wonder what the ratio of spam and junk pages is to human-generated content at the moment?

If they're being overwhelmed, it may be simpler, now, for them first to define what junk is, and bury it, than define what quality is.

If the average Joe can buy software like Traffic Equalizer, goodness knows what black hat progammers can come up with e.g.

I searched on my site name with a keyword I hadn't focussed on yet. Up cames lots of mentions on dud blogspot.com sites, which in turn immediately redirected to something like a simple Adsense block on a bare white page.

Good quality backlinks with unique content on every page is the way to go, I think. Be wary of what CMS systems output, in that regard i.e.

- A directory, with
- 20,000 pages of the same header, footer and sidebars,
- with 100 words of (unique?) text in the middle,
- In a highly competitive market (travel, hotels)
- topped off with link-exchange backlinks to the index page only.

"It's a useful resource!" screeches the de-indexed webbie.
"Yawn" says the SE algorithm, and dumps the lot.

2:24 pm on Jan 19, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Put yourself in a search engine's shoes. Spammers put millions of dud pages on the SE's hard disks. You can define quality by certain criteria, and spam by other.

Just to clarify, I completely support the removal of junk pages from any search engine index. I was responding to the banning of an entire site because some of the pages are considered spam. According to written postings on this board by a Google rep, that IS a policy that Google embraces.

I am just hoping that MSN does not follow suit.

It is like punishing an entire school for the behavior of a few students; or firing an entire office full of people because a couple employees are not competent. It is over-reacting, self-defeating, and should be unacceptable by law, or at the very least, by common sense.

But yes, absolutely, remove all the spam pages -- you'll get no argument from me about that!

....................................

1:05 pm on Jan 20, 2007 (gmt 0)

10+ Year Member



I was banned by Google too. Some of my pages were spammy. If it's a competitive area, the de-indexed sites will be replaced by equally good ones, and some 'near the knuckle' webmasters will get the message.

I'm just waiting to see if the recent GoogleSpasm is temporary, or permanent. Call this one the Alzheimer Update; "we've forgotten some of your pages".

3:46 pm on Jan 20, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Some of my pages were spammy.

I admire your honesty. Unfortunately, there are lots of people who are dropped from a search index for reasons which are unrelated to spamming, and because the powers-that-be refuse to provide any guidance as to why they are suddenly gone, they are left scratching their heads with no recourse. In my opinion, it's disgraceful that multi-billion dollar corporations are willing to occasionally destroy the legitimate small business because they want to catch the spammers in their midst. We all want the spammers eradicated, but not at the expense of the innocent who are simply making mistakes. And this authoritarian over-reaction is avoidable:

-- Remove any & all pages in violation of their rules, but not entire sites (if other pages have quality content and fall within the rules);
-- Provide a mechanism whereby a siteowner can receive a "fair warning" notice that a significant percentage of their pages are about to be removed (the trigger would be, for example, 25%);
-- Provide some (even a minimal) clue as to the root cause of the infraction.
-- Allow the siteowner to fix the problem within a defined time (for example, 14 days). If they do not, it is their choice and they have no one to blame but themselves.

If the major search services were to adopt those 4 reasonable suggestions, almost ALL of the complaints about them would vanish. We can only hope.

.............................................

3:59 pm on Jan 20, 2007 (gmt 0)

10+ Year Member



They can't tell you why, so as to give no clues to spammers, I assume.

Also, legit webmasters can get booted if their site fits the same profile as sites the SE wants gone. Nothing personal.

The simplest thing would be to make different sites for different SE's, and only allow their bots to index them.

5:37 pm on Jan 21, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



To be fair to MSN their crawling seems spot on here.

We revamped a site yesterday. Within 2 hours their robot hit a page that was moved and returned a 404. 10 minutes later their bot was back taking the index page. Overnight and today it spidered the new site taking one or two pages a time - A perfect crawl :)

5:45 pm on Jan 21, 2007 (gmt 0)



No problems here either, if anything MSN seems to do a faster job than Google and Yahoo.

New pages usually crawled within 24 hours, and indexed within 3-5 days.

9:21 am on Jan 22, 2007 (gmt 0)

10+ Year Member



Just to bring this thread back on topic and to clarify my concerns which are based on observations of hundreds of sites I run.

Old sites (pre September ish 2006 have all pages indexed no problem and they are still all included)

New sites from the end of 2006 onwards only get the homepage indexed with no further crawling. To include further pages a manual submit is required which is tedious but works.

This is clearly nothing to do with spam or an algo changed (I am aware of the recent algo change) else the pages would not even be included via a manual submit.

Come on MSNDude without us there is no internet so a swift reply would be much apprecaiated.

10:20 am on Jan 22, 2007 (gmt 0)

5+ Year Member



I have pages waiting to be indexed since September 06...

Another More Detailed reply from MSNDude regarding this issue would be nice...

If this is going to be the norm MSN...so be it...if not, at least look into it. It's going on 5 months and still waiting.
Fish Texas

12:05 am on Jan 29, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Just to bring this thread back on topic and to clarify my concerns which are based on observations of hundreds of sites I run.
Old sites (pre September ish 2006 have all pages indexed no problem and they are still all included)

New sites from the end of 2006 onwards only get the homepage indexed with no further crawling. To include further pages a manual submit is required which is tedious but works.

Just launched a small new site and non index pages are showing up in the index in under 14 days. (No manual submitting etc was done)

Google was the quickest to index it - under 3 days, MSN under 14 days and Yahoo have yet to index it.

3:52 am on Jan 30, 2007 (gmt 0)

5+ Year Member



MSNBot crawling my site like crazy...pages are beginning to be indexed...things are looking up.
Fish Texas
3:39 pm on Jan 30, 2007 (gmt 0)

10+ Year Member



Still only have home pages listed on my sites built since September 2006.

Hopefully it will crawl mine like crazy soon!

7:53 pm on Jan 30, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I have some sites that the bot is hitting more or less normally (once per day or at least once every other day) and some sites that it hits once per week.

But I also have one site that it is hitting twice per day and one site that I cannot get the bot to go to at all no matter what I do (this site has been on the web for a full year and is as carefully designed as the site that the bot hits twice per day).

So I don't know what is going on, and as the other thread here says, the "Live search submission page (is) broken".

.................................