Welcome to WebmasterWorld Guest from

Message Too Old, No Replies

What's the realistic indexing time for new pages?

How fast should a site get crawled and indexed?



3:39 am on Oct 3, 2006 (gmt 0)

WebmasterWorld Senior Member whitey is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

I was wondering how much time members are expecting new pages and sites to take to fully index, assuming that they have a few good links into their sites with the navigation and sitemaps well constructed.

Around 3 months ago we changed our pages from having URL's with underscores to hyphens and put 301 redirects from old to new, plus loaded the URL's into sitemaps. The old pages disappeared into the supplementals, but the new pages have not indexed.

We have a number of large sites with new pages varying in depth from 3 to 6 levels and up to 78k pages. Some of those sites which have identical sitemap pages, and navigation are fully indexed with PR 1-3 at level 5/6.

Others are very slow - with only a depth of level 2 cached, except where there are very PR7 authority / strong links - but even here the crawling is not full .

Shouldn't things have happened by now?

Patrick Taylor

12:29 pm on Oct 3, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

we changed our pages from having URL's with underscores to hyphens and put 301 redirects from old to new

I did something similar to a site some time back, and I wish I'd done it not with 301 redirects but with the old page containing just a link to the new one, with "page moved" or something like that, and then deleting the old pages a few months later.

I can't prove cause and effect, but from then on the site never performed as it previously had. Pages didn't recover their former PR or ranking even though external incoming links were corrected. I put it down to Google being suspicious about the wholesale 301 redirect, and now I use those only sparingly.


12:51 pm on Oct 3, 2006 (gmt 0)

5+ Year Member

I did a similar thing except that I have not (yet) done the 301 redirect from old to new. The supplemental outcome is similar, except that both old and new are in supplemental. That was 4 months ago.

Oddly, 1% of new pages got out of supplemental into the regular index.
There appears to be no reason that the 1% got out.

The supplemental results experts and duplicate content experts indicate that they have observed it taking one year for a full supplemental recovery. Gee whiz.

My only imagination is that a large number of new urls = "Looks close enough to spam for us."


1:01 pm on Oct 3, 2006 (gmt 0)

10+ Year Member

We ran into problems making a similar switch about a year and a half ago. In our case new URLs were indexed pretty quickly, but the old URLs were still hanging around as supplemental, and often outranking the new ones. I made the rash decision to use the URL removal tool on the old URLs. Big mistake. Traffic plunged almost immediately and it was nine months before the new URLs began to regain the positions held by their old equivilants. I suspect there's an aging factor involved for new URLs, possibly amplified by the number of new URLs appearing at the same time.


1:30 pm on Oct 3, 2006 (gmt 0)

10+ Year Member

Indexing seems slipshod.

We have a news service and some pages are indexed fairly quickly.

But if pages are added in sections of the site that are not part of the news service, it can take months.

What is quite amazing is that Google promotes a search service that you can use to have your visitors to your site use to search your site. And then they don't even keep your site information up to date in their search engine by indexing pages.

We pulled it from the site because it just returned oudated and or incomplete information.

Our experience is that you just can't tell at all when, or if, Google will index you.

Patrick Taylor

2:03 pm on Oct 3, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

I've no experience of working with multi-level sites of 78k pages, but I imagine a 301 redirect to new page names in that type of situation would be pretty risky and a big gulp for Google.

On smallish sites, typically, when I add a page, Google crawls it within a week or so and then indexes it within a month - sometimes longer, especially if the page is in a subdirectory. Perhaps it's a resource issue. I've found Yahoo to be very slow to pick up new pages, and MSN to be very very quick.


2:57 pm on Oct 3, 2006 (gmt 0)

5+ Year Member

Oddly, 1% of changed URLs (20,000) got out of supplemental into the regular index within 3 months.

I forgot to mention that all brand new pages (50-100 of them) that have been added since the redesign have been indexed rather quickly.


Featured Threads

Hot Threads This Week

Hot Threads This Month