homepage Welcome to WebmasterWorld Guest from 54.211.97.242
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Microsoft / Bing Search Engine News
Forum Library, Charter, Moderators: mack

Bing Search Engine News Forum

    
MSN now very slow to include all pages
It used to index them all within weeks
mancunian

10+ Year Member



 
Msg#: 3215848 posted 12:13 pm on Jan 11, 2007 (gmt 0)

I have submitted lots of websites over several years to MSN.co.uk and it has always crawled and indexed all of my web pages very quickly ie within weeks.

Since September 2006 or the end of last year from a UK perspective MSN has started to index very slowly and I am seeing 1 page indexed so far for each of my new sites.

I am hoping that in its next cycle (in another month?) MSN will index all the other pages linked to the page I submitted but fear it will just keeping showing 1 page for each site?

Anybody else seeing dramticvally slower inclusion of pages compared to 6 months ago?

 

Fish_Texas

5+ Year Member



 
Msg#: 3215848 posted 3:14 pm on Jan 12, 2007 (gmt 0)

You Bet...
It's taking months compared to weeks last year.
Perhaps the new MSNDude can address this, why don't you post a new thread to attention of new MSNDude.

Fish Texas

Reno

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3215848 posted 12:19 am on Jan 13, 2007 (gmt 0)

I'm seeing the exact same thing. At one point last year I had to post a question in the robots.txt section of this forum for help in SLOWING DOWN the msn bot. Now, I feel like I need to send out a special invitation with flowers & chocolate. Whatever is happening is frustrating and I can only hope it's because they are planning to release a better operating search service in the near future. Time will (presumably) tell.

...............................

mancunian

10+ Year Member



 
Msg#: 3215848 posted 12:22 pm on Jan 15, 2007 (gmt 0)

It seems that submitting every single page one at a time using the MSN submission page works at getting more pages listed fairly quickly (within weeks)

However surely submitting one page (home page) should result in a crawl of all pages linked to from the home page? (as it used to last year)

I have submitted single pages several months ago for several sites and still only one pages is showing per site. It is hard work having to manually submit each and every page!

MSNDude please fix your very very slow crawling or worse still that MSN bot does not even follow same site links anymore!

TravelPixel

5+ Year Member



 
Msg#: 3215848 posted 7:14 pm on Jan 15, 2007 (gmt 0)

ive noticed this with MSN actually over the last two months. It bizarrely indexes some of our less important pages and hasnt touched any of the user generated good stuff. Google seems to index that with no problem.

Would be interesting to know what is happening.

Ezer

5+ Year Member



 
Msg#: 3215848 posted 3:46 am on Jan 16, 2007 (gmt 0)

Maybe MSN and Live are transfering searching functions and cause some bots connection problems.

Sites de-indexed problems appeared more and more recently.

You may try this url to submit sites :: search.live.com/docs/submit.aspx

I think they want you to submit to LIVE.

[edited by: Ezer at 3:47 am (utc) on Jan. 16, 2007]

msndude

10+ Year Member



 
Msg#: 3215848 posted 4:27 am on Jan 16, 2007 (gmt 0)

Just so this is clear -- we only have one search engine and it's Live Search.

Live Search simply refers to the search engine as it has existed since the release in September, prior to which it was referred to as MSN Search.

Our one bot is MSNBot, and I'm not aware of anything new that might result in slow indexing. There's always some variance in terms of time for this.

fjpapaleo

10+ Year Member



 
Msg#: 3215848 posted 11:47 am on Jan 16, 2007 (gmt 0)

"I'm not aware of anything new that might result in slow indexing"

Dude, my site is two years old. Yahoo has 120,938 pages indexed, MSN has 94! No problem?

mancunian

10+ Year Member



 
Msg#: 3215848 posted 2:53 pm on Jan 17, 2007 (gmt 0)

MSNDude your bot is broken without a doubt I have written hundreds of websites over many years and all pages get indexed within weeks until the last few months.....

Only the page I manually submit gets indexed and no further crawl or indexing of the other pages linked to from that page are indexed even after several months.

I can get them all included if I manually submit one at a time but surely your bot should crawl links from a homepage and include them as it used to last year.

This cannot be due to an algo change as I can get the pages in if submit each one individually.

I am submitting here [search.msn.co.uk...]

Thanks for reading this thread and taking the trouble to reply.

mancunian

10+ Year Member



 
Msg#: 3215848 posted 3:04 pm on Jan 17, 2007 (gmt 0)

PS Google and Yahoo are indexing these sites and all pages no problem.

Reno

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3215848 posted 7:24 pm on Jan 18, 2007 (gmt 0)

There is no question in my mind that something about the behavior of the msnbot has changed over the past several months. Sites of ours that it had fully indexed are now gone, other sites are included but are only partially indexed. So something has happened, as the observations in this thread confirm.

ps. I have had limited success in getting pages back into MSN by submitting them one at a time, but have also found that doing that slow & tedious process is still NOT restoring (so far at least) many of the pages that used to be included.

Also, I hasten to add that NO significant changes had been made to any of those pages that are now missing -- one day they were in the index, the next day gone.

...........................

tigertom

10+ Year Member



 
Msg#: 3215848 posted 11:53 pm on Jan 18, 2007 (gmt 0)

The reason is probably that MSN got complaints from webmasters about the MSNbot being too greedy. So now it's very tentative with its spidering.

Read it on a site while I was researching robots.txt tutorials.

And no, I can't remember the site name.

Reno

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3215848 posted 1:32 am on Jan 19, 2007 (gmt 0)

The reason is probably that MSN got complaints from webmasters about the MSNbot being too greedy. So now it's very tentative with its spidering.

That may explain some of what is going on. Still, it is curious that once well-indexed sites are dropping out entirely, or, the number of pages is reduced from close to 100% down to 5-10%. I still have big hopes that "Live" will eventually become more of a competitor to Google, so the sooner that they get back on track the better it will be for them and for us.

.....................................

tigertom

10+ Year Member



 
Msg#: 3215848 posted 1:59 am on Jan 19, 2007 (gmt 0)

They are also banning sites. This happened to me. Search on threads I've posted in recently for the possible causes.

Reno

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3215848 posted 3:17 am on Jan 19, 2007 (gmt 0)

They are also banning sites

If what you say is true then that would mean that instead of trying to develop a search engine that brings back the highest quality results based on an algo that is truly superior to Google, they would be intending to follow Google's example and start punishing sites OUT of their database, which of course will mean even less of an index from which to find the best possible results. I really hope that is not the case -- the web needs a viable alternative to Google, not a pale imitation.

...............................

tigertom

10+ Year Member



 
Msg#: 3215848 posted 12:41 pm on Jan 19, 2007 (gmt 0)

I disagree.

Put yourself in a search engine's shoes. Spammers put millions of dud pages on the SE's hard disks. You can define quality by certain criteria, and spam by other.

You'd be happy to set a filter that deletes most spam, with some collateral damage. So some webmasters whine, so what? If your SE is tanking, like MSN and Yahoo, that's not a priority.

I wonder what the ratio of spam and junk pages is to human-generated content at the moment?

If they're being overwhelmed, it may be simpler, now, for them first to define what junk is, and bury it, than define what quality is.

If the average Joe can buy software like Traffic Equalizer, goodness knows what black hat progammers can come up with e.g.

I searched on my site name with a keyword I hadn't focussed on yet. Up cames lots of mentions on dud blogspot.com sites, which in turn immediately redirected to something like a simple Adsense block on a bare white page.

Good quality backlinks with unique content on every page is the way to go, I think. Be wary of what CMS systems output, in that regard i.e.

- A directory, with
- 20,000 pages of the same header, footer and sidebars,
- with 100 words of (unique?) text in the middle,
- In a highly competitive market (travel, hotels)
- topped off with link-exchange backlinks to the index page only.

"It's a useful resource!" screeches the de-indexed webbie.
"Yawn" says the SE algorithm, and dumps the lot.

Reno

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3215848 posted 2:24 pm on Jan 19, 2007 (gmt 0)

Put yourself in a search engine's shoes. Spammers put millions of dud pages on the SE's hard disks. You can define quality by certain criteria, and spam by other.

Just to clarify, I completely support the removal of junk pages from any search engine index. I was responding to the banning of an entire site because some of the pages are considered spam. According to written postings on this board by a Google rep, that IS a policy that Google embraces.

I am just hoping that MSN does not follow suit.

It is like punishing an entire school for the behavior of a few students; or firing an entire office full of people because a couple employees are not competent. It is over-reacting, self-defeating, and should be unacceptable by law, or at the very least, by common sense.

But yes, absolutely, remove all the spam pages -- you'll get no argument from me about that!

....................................

tigertom

10+ Year Member



 
Msg#: 3215848 posted 1:05 pm on Jan 20, 2007 (gmt 0)

I was banned by Google too. Some of my pages were spammy. If it's a competitive area, the de-indexed sites will be replaced by equally good ones, and some 'near the knuckle' webmasters will get the message.

I'm just waiting to see if the recent GoogleSpasm is temporary, or permanent. Call this one the Alzheimer Update; "we've forgotten some of your pages".

Reno

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3215848 posted 3:46 pm on Jan 20, 2007 (gmt 0)

Some of my pages were spammy.

I admire your honesty. Unfortunately, there are lots of people who are dropped from a search index for reasons which are unrelated to spamming, and because the powers-that-be refuse to provide any guidance as to why they are suddenly gone, they are left scratching their heads with no recourse. In my opinion, it's disgraceful that multi-billion dollar corporations are willing to occasionally destroy the legitimate small business because they want to catch the spammers in their midst. We all want the spammers eradicated, but not at the expense of the innocent who are simply making mistakes. And this authoritarian over-reaction is avoidable:

-- Remove any & all pages in violation of their rules, but not entire sites (if other pages have quality content and fall within the rules);
-- Provide a mechanism whereby a siteowner can receive a "fair warning" notice that a significant percentage of their pages are about to be removed (the trigger would be, for example, 25%);
-- Provide some (even a minimal) clue as to the root cause of the infraction.
-- Allow the siteowner to fix the problem within a defined time (for example, 14 days). If they do not, it is their choice and they have no one to blame but themselves.

If the major search services were to adopt those 4 reasonable suggestions, almost ALL of the complaints about them would vanish. We can only hope.

.............................................

tigertom

10+ Year Member



 
Msg#: 3215848 posted 3:59 pm on Jan 20, 2007 (gmt 0)

They can't tell you why, so as to give no clues to spammers, I assume.

Also, legit webmasters can get booted if their site fits the same profile as sites the SE wants gone. Nothing personal.

The simplest thing would be to make different sites for different SE's, and only allow their bots to index them.

gpmgroup

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3215848 posted 5:37 pm on Jan 21, 2007 (gmt 0)

To be fair to MSN their crawling seems spot on here.

We revamped a site yesterday. Within 2 hours their robot hit a page that was moved and returned a 404. 10 minutes later their bot was back taking the index page. Overnight and today it spidered the new site taking one or two pages a time - A perfect crawl :)

bobothecat



 
Msg#: 3215848 posted 5:45 pm on Jan 21, 2007 (gmt 0)

No problems here either, if anything MSN seems to do a faster job than Google and Yahoo.

New pages usually crawled within 24 hours, and indexed within 3-5 days.

mancunian

10+ Year Member



 
Msg#: 3215848 posted 9:21 am on Jan 22, 2007 (gmt 0)

Just to bring this thread back on topic and to clarify my concerns which are based on observations of hundreds of sites I run.

Old sites (pre September ish 2006 have all pages indexed no problem and they are still all included)

New sites from the end of 2006 onwards only get the homepage indexed with no further crawling. To include further pages a manual submit is required which is tedious but works.

This is clearly nothing to do with spam or an algo changed (I am aware of the recent algo change) else the pages would not even be included via a manual submit.

Come on MSNDude without us there is no internet so a swift reply would be much apprecaiated.

Fish_Texas

5+ Year Member



 
Msg#: 3215848 posted 10:20 am on Jan 22, 2007 (gmt 0)

I have pages waiting to be indexed since September 06...

Another More Detailed reply from MSNDude regarding this issue would be nice...

If this is going to be the norm MSN...so be it...if not, at least look into it. It's going on 5 months and still waiting.
Fish Texas

gpmgroup

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3215848 posted 12:05 am on Jan 29, 2007 (gmt 0)

Just to bring this thread back on topic and to clarify my concerns which are based on observations of hundreds of sites I run.
Old sites (pre September ish 2006 have all pages indexed no problem and they are still all included)

New sites from the end of 2006 onwards only get the homepage indexed with no further crawling. To include further pages a manual submit is required which is tedious but works.

Just launched a small new site and non index pages are showing up in the index in under 14 days. (No manual submitting etc was done)

Google was the quickest to index it - under 3 days, MSN under 14 days and Yahoo have yet to index it.

Fish_Texas

5+ Year Member



 
Msg#: 3215848 posted 3:52 am on Jan 30, 2007 (gmt 0)

MSNBot crawling my site like crazy...pages are beginning to be indexed...things are looking up.
Fish Texas

mancunian

10+ Year Member



 
Msg#: 3215848 posted 3:39 pm on Jan 30, 2007 (gmt 0)

Still only have home pages listed on my sites built since September 2006.

Hopefully it will crawl mine like crazy soon!

Reno

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3215848 posted 7:53 pm on Jan 30, 2007 (gmt 0)

I have some sites that the bot is hitting more or less normally (once per day or at least once every other day) and some sites that it hits once per week.

But I also have one site that it is hitting twice per day and one site that I cannot get the bot to go to at all no matter what I do (this site has been on the web for a full year and is as carefully designed as the site that the bot hits twice per day).

So I don't know what is going on, and as the other thread here says, the "Live search submission page (is) broken".

.................................

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Microsoft / Bing Search Engine News
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved