Forum Moderators: mack
Since September 2006 or the end of last year from a UK perspective MSN has started to index very slowly and I am seeing 1 page indexed so far for each of my new sites.
I am hoping that in its next cycle (in another month?) MSN will index all the other pages linked to the page I submitted but fear it will just keeping showing 1 page for each site?
Anybody else seeing dramticvally slower inclusion of pages compared to 6 months ago?
Fish Texas
...............................
However surely submitting one page (home page) should result in a crawl of all pages linked to from the home page? (as it used to last year)
I have submitted single pages several months ago for several sites and still only one pages is showing per site. It is hard work having to manually submit each and every page!
MSNDude please fix your very very slow crawling or worse still that MSN bot does not even follow same site links anymore!
Would be interesting to know what is happening.
Sites de-indexed problems appeared more and more recently.
You may try this url to submit sites :: search.live.com/docs/submit.aspx
I think they want you to submit to LIVE.
[edited by: Ezer at 3:47 am (utc) on Jan. 16, 2007]
Live Search simply refers to the search engine as it has existed since the release in September, prior to which it was referred to as MSN Search.
Our one bot is MSNBot, and I'm not aware of anything new that might result in slow indexing. There's always some variance in terms of time for this.
Dude, my site is two years old. Yahoo has 120,938 pages indexed, MSN has 94! No problem?
Only the page I manually submit gets indexed and no further crawl or indexing of the other pages linked to from that page are indexed even after several months.
I can get them all included if I manually submit one at a time but surely your bot should crawl links from a homepage and include them as it used to last year.
This cannot be due to an algo change as I can get the pages in if submit each one individually.
I am submitting here [search.msn.co.uk...]
Thanks for reading this thread and taking the trouble to reply.
ps. I have had limited success in getting pages back into MSN by submitting them one at a time, but have also found that doing that slow & tedious process is still NOT restoring (so far at least) many of the pages that used to be included.
Also, I hasten to add that NO significant changes had been made to any of those pages that are now missing -- one day they were in the index, the next day gone.
...........................
Read it on a site while I was researching robots.txt tutorials.
And no, I can't remember the site name.
The reason is probably that MSN got complaints from webmasters about the MSNbot being too greedy. So now it's very tentative with its spidering.
.....................................
They are also banning sites
...............................
Put yourself in a search engine's shoes. Spammers put millions of dud pages on the SE's hard disks. You can define quality by certain criteria, and spam by other.
You'd be happy to set a filter that deletes most spam, with some collateral damage. So some webmasters whine, so what? If your SE is tanking, like MSN and Yahoo, that's not a priority.
I wonder what the ratio of spam and junk pages is to human-generated content at the moment?
If they're being overwhelmed, it may be simpler, now, for them first to define what junk is, and bury it, than define what quality is.
If the average Joe can buy software like Traffic Equalizer, goodness knows what black hat progammers can come up with e.g.
I searched on my site name with a keyword I hadn't focussed on yet. Up cames lots of mentions on dud blogspot.com sites, which in turn immediately redirected to something like a simple Adsense block on a bare white page.
Good quality backlinks with unique content on every page is the way to go, I think. Be wary of what CMS systems output, in that regard i.e.
- A directory, with
- 20,000 pages of the same header, footer and sidebars,
- with 100 words of (unique?) text in the middle,
- In a highly competitive market (travel, hotels)
- topped off with link-exchange backlinks to the index page only.
"It's a useful resource!" screeches the de-indexed webbie.
"Yawn" says the SE algorithm, and dumps the lot.
Put yourself in a search engine's shoes. Spammers put millions of dud pages on the SE's hard disks. You can define quality by certain criteria, and spam by other.
I am just hoping that MSN does not follow suit.
It is like punishing an entire school for the behavior of a few students; or firing an entire office full of people because a couple employees are not competent. It is over-reacting, self-defeating, and should be unacceptable by law, or at the very least, by common sense.
But yes, absolutely, remove all the spam pages -- you'll get no argument from me about that!
....................................
I'm just waiting to see if the recent GoogleSpasm is temporary, or permanent. Call this one the Alzheimer Update; "we've forgotten some of your pages".
Some of my pages were spammy.
-- Remove any & all pages in violation of their rules, but not entire sites (if other pages have quality content and fall within the rules);
-- Provide a mechanism whereby a siteowner can receive a "fair warning" notice that a significant percentage of their pages are about to be removed (the trigger would be, for example, 25%);
-- Provide some (even a minimal) clue as to the root cause of the infraction.
-- Allow the siteowner to fix the problem within a defined time (for example, 14 days). If they do not, it is their choice and they have no one to blame but themselves.
If the major search services were to adopt those 4 reasonable suggestions, almost ALL of the complaints about them would vanish. We can only hope.
.............................................
Also, legit webmasters can get booted if their site fits the same profile as sites the SE wants gone. Nothing personal.
The simplest thing would be to make different sites for different SE's, and only allow their bots to index them.
We revamped a site yesterday. Within 2 hours their robot hit a page that was moved and returned a 404. 10 minutes later their bot was back taking the index page. Overnight and today it spidered the new site taking one or two pages a time - A perfect crawl :)
New pages usually crawled within 24 hours, and indexed within 3-5 days.
Old sites (pre September ish 2006 have all pages indexed no problem and they are still all included)
New sites from the end of 2006 onwards only get the homepage indexed with no further crawling. To include further pages a manual submit is required which is tedious but works.
This is clearly nothing to do with spam or an algo changed (I am aware of the recent algo change) else the pages would not even be included via a manual submit.
Come on MSNDude without us there is no internet so a swift reply would be much apprecaiated.
Another More Detailed reply from MSNDude regarding this issue would be nice...
If this is going to be the norm MSN...so be it...if not, at least look into it. It's going on 5 months and still waiting.
Fish Texas
Just to bring this thread back on topic and to clarify my concerns which are based on observations of hundreds of sites I run.
Old sites (pre September ish 2006 have all pages indexed no problem and they are still all included)New sites from the end of 2006 onwards only get the homepage indexed with no further crawling. To include further pages a manual submit is required which is tedious but works.
Just launched a small new site and non index pages are showing up in the index in under 14 days. (No manual submitting etc was done)
Google was the quickest to index it - under 3 days, MSN under 14 days and Yahoo have yet to index it.
Hopefully it will crawl mine like crazy soon!
But I also have one site that it is hitting twice per day and one site that I cannot get the bot to go to at all no matter what I do (this site has been on the web for a full year and is as carefully designed as the site that the bot hits twice per day).
So I don't know what is going on, and as the other thread here says, the "Live search submission page (is) broken".
.................................
