New Content Crawling - why would it take months?

[nmfam]

We're wondering why it has been taking so long to get the content crawled on a number of our sites. Sites are all reasonably shallow linked (not too many clicks to each page) and we have sitemap files setup and linked from our Google Webmaster account, so I thought it would go quickly.

We have one site in particular has 121 in the sitemap and according to Google Webmaster Central, only 57 of those pages have been crawled ... and we've been waiting for *months*! Now granted, there was some 404 issue for a while but that was resolved 6 weeks ago. Still waiting. :(

We have another new site that has been live with 234 new pages of content ... all live since Nov 1. 14 pages are indexed according to Google insite: tool.

At this rate, it will be a some time in mid 2012 when the content is all finally recognized by Google..

Any thoughts?

[Yellow_Sun]

nmfam,

You have answered some of your questions; shallow linked, new website, 404 issues, etc.

With newer websites it takes time to get crawled fully and trust needs to be built with Google. Great links will help do a deeper crawl as well, but you will need to have some patience. While Google is doing its thing, I would continue adding new unique quality content and obtaining quality links as much as time allows. Doing so will definitely speed things up, but either way you have a ways to go yet until all pages will get indexed.

Keep working at it and quality over quantity.

[scottsonline]

Hopefully once google caffeine is released indexing speed of sites will pick up.

[nmfam]

> "You have answered some of your questions; shallow linked, new website, 404 issues, etc."

Yes but those issues were resolved months ago. Still, very little activity since these were fixed. Should it really take months for 150 pages to be discovered? It doesn't "feel" right to me. I don't see anything else for me to fix though.

I also have another site with 120 pages of content ... its been stuck on 6 pages indexed for about 3-4 months. No visible issues for me to fix.



> "I would continue adding new unique quality content and obtaining quality links as much as time allows."

But what is the value of adding new content if it never spiders the existing content? I mean, by adding new content, will it encourage Google bot to come around a little faster somehow?




> "Hopefully once google caffeine is released indexing speed of sites will pick up."

Oh, I thought that was released back in late Spring of 2010? Is it still not fully out there?

[tedster]

Are you taking a shot at Caffeine, scottslonline? ;)

Just so there's no confusion for anyone - Caffeine was fully rolled out in June 2010 reference [webmasterworld.com].

[aristotle]

nmfam - Do you mean that these pages haven't been crawled, or do you mean that they haven't been indexed?

[Mark_A]

nmfam have you checked your robots.txt - I know it does not seem likely but bear with me.

I had a site with slow indexing problems, (it took me up to two months to get new content indexed) I checked the robots.txt and could see that various directories (datasheets mainly) were being excluded. I changed the robots.txt to permit these directories and the crawl rate immediately flew up and then the indexing issue seemed to go away.

[Shaddows]

Go get some links to deep pages. Nothing aids crawling as much as an inbound link.

Apart from that, its just vague generalities like
- Nav structure
- Inline linking
- Semantic cohesion

And for proactive traffic acquisition
- Social Media (get a twitter and facebook account)
- Link Development [webmasterworld.com]

Added- Server logs. Have you been crawled but not indexed, or simply not crawled. If you're being crawled (especially if crawled frequently), its likely Google has decided there is insufficient value to those pages to warrant inclusion in the main index. You might want to search for not-quite-exact phrases on the pages to see if they are 'supplemental'. Ultimately though, you will need to increase the perceived value by enriching the content (Google validates) or getting new links (external validation).

[edited by: Shaddows at 2:43 pm (utc) on Dec 7, 2010]

[jimbeetle]

Great links will help do a deeper crawl as well...

Yep, PR is the most important factor in G's crawl algo. More good links equal more crawl budget and faster indexing. You might also want to consider using social stuff such as Twitter for URL discovery as in Google Confirms - Social Media Signals Are Used in Some Search Results [webmasterworld.com].

[HuskyPup]

Any thoughts?

Yep, Google's just about almost broken on many fronts!

As a matter of interest how are you looking in Bing, Yahoo! and Duck Duck Go?

[tangor]

One other aspect often overlooked is duplicate meta statements on pages. Easy to overlook on templated sites. If these look the same, google (or most SEs) will not bother.

[nmfam]

Ok, here are my findings:

1. How I diagnosed the problem:
According to Google webmaster central, only 57 of the 129 pages in my sitemap were indexed. But if I do "site:domain.com", tf shows 321 pages are actually there...ironically many of the pages are double-indexed; first 13 pages are all the semantic URLs and the subsequent pages are the unmasked URLs ... not sure how the heck Google found those. So, I guess I need to 301 redirect if I detect that the masked semantic URL isn't used? Regardless, neither version of any of the content actually ranks for anything in the serps.

2. Bing and Yahoo:
In both these cases, I can run "site:domain.com" and I find the pages are all there. Again though, not showing up for anything. Yahoo in particular won't even return us in the serps for a search of "domain.com", although ironically it does return us as #3 when I search simply for "domain". Ultimately though we're getting no traffic from anyone - excellent root-word domain name, 150 pages of unqiue article quality content, and about 10-15 visitors only each day. :( Note - neither Bing or Yahoo show the non-semantic content that Google somehow found.


3. Robots.txt is pretty clean this is it:
User-agent: *
Sitemap: /index.php/sitemap.xml


4. Duplicate Meta Statements:
This is a good point. Our title tags are all unique, but most of our title tags are the same. Can this really be such a big issue? Do I need to take the time now to write a meta desc for 100s of pages or can I just remove the meta desc?

5. Blog Tags:
We're using a modified WordPress framework for a CMS - I wonder if some of the meta tags can cause an issue:

<link rel="alternate" type="application/rss+xml" title="RSS 2.0" href="/feed/" />
<link rel="alternate" type="text/xml" title="RSS .92" href="/feed/rss/" />
<link rel="alternate" type="application/atom+xml" title="Atom 0.3" href="/feed/atom/" />
<link rel="pingback" href="/xmlrpc.php" />

[tedster]

Unique meta descriptions used to be very important for getting more pages in the main index - but that was before Caffeine was deployed. I haven't tested it since then - but my sense of it is things have shifted.

I do think it would be better not to have any meta descriptions at all than to serve a whole lot of duplicates. But especially at the scale you are describing, you're probably much better off the actually write unique descriptions. It's not that big a job, just big enough to be a bit of a PITA.

[jimbeetle]

<link rel="alternate" type="application/rss+xml" title="RSS 2.0" href="/feed/" />
<link rel="alternate" type="text/xml" title="RSS .92" href="/feed/rss/" />
<link rel="alternate" type="application/atom+xml" title="Atom 0.3" href="/feed/atom/" />
<link rel="pingback" href="/xmlrpc.php" />

No, you're grasping at straws, those are necessary for the feeds to work correctly.

And truly, you can analyze and diagnose this and that and something else, but as stated earlier, it all comes down to links. So to cut to the chase, what's the site's backlink profile like?

Do I need to take the time now to write a meta desc for 100s of pages or can I just remove the meta desc?

Please be working in one of my niches ;-)

The description is the one and only place where you have an opportunity to expand on the title element and tell a possible visitor what the page is about before they click. Descriptions aren't always used by the SEs in snippets but sometimes are (usually if there is a direct query to text match). Well-crafted titles and descriptions can work together to "get the click" even if it is further down the SERPs. It is well worth spending the time to do it.

[scottsonline]

Tedster yes...I was grumpy post jets mauling.

Caffeine is one of those things that worked in the sandbox better than it has in the wild. It too is partly responsible for this mess.

[tangor]

I removed meta statements from one elder site and made sure all the title element and the actual page titles were unique (to each of about 1,700 pages) and saw a more consistent crawl and SERP returns from all SEs... an experiment begun in 2002. Worked well enough that these days I don't bother with meta statements on new sites.

[brinked]

Unique meta descriptions used to be very important for getting more pages in the main index - but that was before Caffeine was deployed. I haven't tested it since then - but my sense of it is things have shifted.

I do think it would be better not to have any meta descriptions at all than to serve a whole lot of duplicates. But especially at the scale you are describing, you're probably much better off the actually write unique descriptions. It's not that big a job, just big enough to be a bit of a PITA.

Google says they do not use meta descriptions to factor in rankings. Either way, this is what people see in google that makes them either click or not click your site. This is your sales pitch to entice potential visitors, so make it count, a great meta description can go a very long way.

I would advise against automating it. WHat I found is the easiest way to go about this is to setup a instant edit front end solution using jquery this way I can go through my sites front end, click "edit" and put in a detailed description for that page that I want people to see in the search engines. This can also be useful for page titles if you have a lot of pages on your site that are generated from a script that you want to customize as easily as possible.

[ken_b]

I'm seeing a bit ofa change for the better in indexing new content.

Pages I uploaded 3 days ago are indexed. Earlier this year I was more often just seeing the sub-index page that linked to them show up when I searched for them, and even that took longer.

The result is that the searcher gets taken to the actual destination page directly from the serps instead of landing 1 level up and having to click thru from there.

(Hopefully this will continue.)

[tedster]

Worked well enough that these days I don't bother with meta statements on new sites.

Google is gang-busters right now rewriting titles and generating their own snippets, ignoring meta-descriptions. The rewriting is query specific, so you can still get some "control" over how the listing looks for your most targeted query terms, but it's nothing like it used to be.

It seems clear that title, meta description and H1 are some of the high-level shards that Google indexes as back end meta-data in their system. If these elements diverge too far from what their data says makes a good result, your expressed preferences will not be honored. That said, I was explroing this rewriting a good bit today, and often Imust admit Google's rewrites do make sense.

Having these three elements as unique on your page is probably still a crawling/indexing help - but most of all it seems to take PR.

[Mark_A]

Google says they do not use meta descriptions to factor in rankings. Either way, this is what people see in google that makes them either click or not click your site. This is your sales pitch to entice potential visitors, so make it count, a great meta description can go a very long way.

They say that, yet they often use a good meta description in their own SERPS to describe a page, and if a searched for keyword appears in the snippet of meta description they have shown, they bold it..

I think there is reason to feel google does use meta descriptions, so we should too.

[HuskyPup]

I think there is reason to feel google does use meta descriptions, so we should too.

I agree plus it does not take that much longer to create a unique description and who's to say it will not be used in the future by ANY search engine?

[nmfam]

Its just ... a LOT of work to go back and do all that. Sure, if we assume our time is worth $0 then why not add those in there. But when limited by time and $$$, have to prioritize things.

[tangor]

I've always figured that $0 (sweat) in gets $0 (return) out...

Every once in a while we have to go back and fix a bad choice (hopefully caught fairly early) and then do the RIGHT THING from then on out.

No lecture, just an observation of how things work.

[jimbeetle]

Its just ... a LOT of work to go back and do all that.

He, he. Again, I hope you're playing in on of my spaces ;-)

As competitive as the web is you simply cannot give other folks an advantage, else you're starting out at a disadvantage, behind 10 lengths out of the gate. That makes it much harder to catch up, especially for a new site.

And while we're at it, if you ask for help and advice you must be prepared to answer questions so people can get a better handle on things and can offer better advice.

So, what's the site's backlink profile like?

[nmfam]

> "He, he. Again, I hope you're playing in on of my spaces ;-) "

If it costs $1.00 of effort and I only make back $0.50, what's the point? This is what I'm saying... I'd rather not chase every silly little thing on all of our websites. Google is getting smarter and frankly I wonder how much some of these things matter any more. We can obsess over the menutia, or we can allocate those same dollars and hours to the things that really matter. That's where I'm coming from.

But hell, if this is why we're not getting crawled, or at least a viable theory as to why (sounds viable), I'm definitely willing to try it! I'm just so tired of chasing my tail on SEO and not seeing the ROI for most of it.

[ken_b]

If it costs $1.00 of effort and I only make back $0.50, what's the point?

The point is that you put in $1.00 worth of effort today as a one time investment and the continually reap the benefit month after month after month, etc.

So sure, you might not get to a positive ROI the next day, but in the long run it could pay off really well.

[Robert Charlton]

Sites are all reasonably shallow linked (not too many clicks to each page)...

Not necessarily a good thing... This suggests to me that, in order to get the shallow or flat structure, you might have built some very large pages with a huge number of nav links on them. This might result in a miniscule amount of link juice available for each nav link.

Do you have a lot of nav links on a page or not? How many?

Directly related to this is what kind of inbound links you have. What is the inbound link quality like? Though a few comments have been made about getting good links and links to deep pages, I'm not seeing any response on this topic. IMO, it's probably the core issue.

[jimbeetle]

That's kind of like my buck a day theory: If I spend x time today to make improvements, add content, whatever, that brings in $1 (or quarter, or half a buck) each subsequent day, then I'm ahead. Rinse and repeat, further ahead.

And nmfam, the unique meta descriptions won't help in getting crawled, but they might help to differentiate the pages to help get them get indexed more quickly.

What will help the crawl rate is backlinks. So? ;-)

[nmfam]

> "The point is that you put in $1.00 worth of effort today as a one time investment and the continually reap the benefit month after month after month, etc."

I'd temper that statement with "maybe" not "will". ;-) Maybe that one thing will be the magic thing that changes things. Or maybe its such minutia that isn't even necessary to rank well anymore, that you just wasted a Saturday afternoon. Or maybe it works out for the next 2 months until Google changes their algo again.

Sorry to sound cynical, but I've wasted a LOT of time and money in the past doing SEO by the book and probably yielding $10-12 per hour for my efforts along the way. We've even hired a well-known SEO company previously, that (after 6 months of high cost monthly fees) ultimately accomplished (wait for it) ... nothing!

I realize this is off topic. And I do appreciate the thoughts of what I could possibly do to get Google to come take a look at this site. Just trying to explain my last point and why I'm skeptical of the idea that we should do anything and everything, always. :)

[Robert Charlton]

nmfam - Sorry to sound cynical as well, but you keep avoiding the question of backlinks.

What kind of inbound links do you have?

And, back to square one... a question that should have preceded the linking question... is your content unique enough, and of high enough quality, that it's worth linking to?