New Content Crawling - why would it take months?

[nmfam]

We're wondering why it has been taking so long to get the content crawled on a number of our sites. Sites are all reasonably shallow linked (not too many clicks to each page) and we have sitemap files setup and linked from our Google Webmaster account, so I thought it would go quickly.

We have one site in particular has 121 in the sitemap and according to Google Webmaster Central, only 57 of those pages have been crawled ... and we've been waiting for *months*! Now granted, there was some 404 issue for a while but that was resolved 6 weeks ago. Still waiting. :(

We have another new site that has been live with 234 new pages of content ... all live since Nov 1. 14 pages are indexed according to Google insite: tool.

At this rate, it will be a some time in mid 2012 when the content is all finally recognized by Google..

Any thoughts?

[tangor]

I agree with Robert and Jim... without more info we can't keep guessing (though I did guess on one thing). Can't post a link, but you can give insight into site structure (links, etc.) and content quality. Thin sites MFA generally don't do well for indexing purposes.

Also, there is no guarantee any SE will index all site pages or, if they do, show results in the SERPS. Secondary there may be other factors involved such as poor URI, duplicate content, template errors, etc. That said, as you said:

At this rate, it will be a some time in mid 2012 when the content is all finally recognized by Google..

Let us know in 2012.

[nmfam]

Inbound links - we haven't made any great effort there yet because we determined there was a problem early on,seeing that Google wasn't even sucking up our content in a predictable way. We've probably spent a couple hundred dollars getting paid blogs and a few press releases ... maybe 10-12 links in total, all to the homepage.

All the articles are 400+ words, uniquely written by our freelance team. The content quality should be relatively high. I can't say that we're curing world hunger, but they are quality articles though.

[HuskyPup]

I've wasted a LOT of time and money in the past doing SEO by the book

Whose book?

I construct all my sites through my proven experience from the early 90s and with working with SE company algos. Anything you need to know and learn can be found at WebmasterWorld for free, there is no necessity to hire in an SEO company unless you need me:-))...hey, I'm very busy with my own stuff!

I have this feeling we're not getting the full picture.

[nmfam]

> "Whose book? "

1. Webmasterworld (I was a member for years under a diff't name)
2. SenSEO
3. Arelis IBP
...all white hat stuff

> "
I have this feeling we're not getting the full picture. "

Tell me about it - I can't figure it out! Well, I'm hoping it is just the meta tag thing. Its all I can see as the issue. Unless perhaps the domain had an issue prior to us acquiring it. :-\

[Shaddows]

According to Google webmaster central, only 57 of the 129 pages in my sitemap were indexed. But if I do "site:domain.com", tf [shows 321 pages are actually there...ironically many of the pages are double-indexed; first 13 pages are all the semantic URLs

Right, so you have almost 130 pages indexed, out of a possible 129.

So the problem is value. You need to add value. Quoting myself:

If you're being crawled (especially if crawled frequently), its likely Google has decided there is insufficient value to those pages to warrant inclusion in the main index... Ultimately though, you will need to increase the perceived value by enriching the content (Google validates) or getting new links (external validation).

Some big-hitters here telling you to
a) Increase links
b) Sort your duplicate problem out

I can add

c) Sort your canonical problem
and
d) If you're going to ask for advice, and it comes back pretty unanimous, you can either ignore it or accept it, but there's not a lot of point claiming its "too much work"

Also, you seem to be aware of the dangers of "chasing the algo", yet you shun the relatively fundamental point of writing unique metas. That's about usability, and even if it isn't used as a page-factor, unique meta-description is probably a sign of overall site quality. Or the converse, non-unique metas are a sign of someone who doesn't want to put the effort in.

BTW, 12 backinks to the homepage on a flat site is a problem, probably more than the metas. You need semantic links to other pages for a start, and more links overall in general.

[nmfam]

UPDATE: Ok this is interesting! I was just about to start working on those meta tag updates this morning. I logged on to Google Webmaster Central just to check first, if any new progress was made on indexing the content from our sitemap XML. Not only was there no progress .. the number of indexed pages (of 129 total) actually went BACKWARDS this week, from 57... to 46.

Now I'm starting to think there's something going on. Thoughts?

PS - I didn't say I was opposed to trying what was being offered and I'm absolutely trying whatever I can now. I just meant by policy, it seems odd to obsess over the details as one member was advocating. I'd have to be full time dedicated to SEO I guess, to have the time for it. Otherwise I must pick my battles - was my only point there. :-\

[tedster]

The webmaster tools numbers are a confirmed bug - ignore those numbers for now. reference [webmasterworld.com]

[jimbeetle]

I just meant by policy, it seems odd to obsess over the details as one member was advocating. I'd have to be full time dedicated to SEO I guess

Not really, not if you have a good content creation workflow. Crafting titles and descriptions should be a part of that process and not something you should have to go back and do. Sure, review them before they go live, but they should be part and parcel of the delivered content.

And obsessing over the details is simply part of SEO 101, a solid base from which to go forward. Everybody giving advice in this thread has made the mistakes -- myself many of them -- and learned from them. We're simply trying to pass on best practices and give you a leg up.