New pages indexed, de-indexed, re-indexed - is this normal?

[topranker]

Most new pages seem to get indexed within 24 hours, then around 40-50% of them get de-indexed again for about 2-14 days then get re-indexed again.

After this process the page is then very stable and is very unlikely to get de-indexed although may drop in rankings as they get older.

Is this normal behavour for all sites as it seems to happen all the time on our site.

The site is 2 years old has plenty of backlinks, hosted in the uk and is a uk site.

Lee

[majestic12]

Just to let you folks know..

I posted uploaded another page... my lost page was re-indexed and ranked as before and the new page is ranking well..

Regards David

[Robert Charlton]

Just to let you folks know..

I posted uploaded another page... my lost page was re-indexed and ranked as before and the new page is ranking well..

David - Thanks for the follow-up. It sounds like this was just a situation of Google databases getting synched up.

It's very possible, even with established pages, to see two different rankings simultneously on different machines. There are constant adjustments being made at datacenters, and it takes time for data to migrate. That said, you may yet see the drop that others are discussing. ;)

Re the second page you uploaded... if it was substantially a dupe of the first page... or if it, say, had the same title... one of those pages is likely to disappear. You can target a phrase on more than one page, but if you use the same content on more than one url, you will have dupe content problems.

[Receptional Andy]

I notice that the original post was from the UK, and that's where I got most of my data from, so I don't know if there's a regional influence on this.

From the sample I took, around 60% dropped shortly after the initial appearance in results. A page with external links or significant internal links was significantly less likely to drop out, but this did not explain the whole relationship between the page and the chance of disappearing.

Note that I did not take a particularly large sample to obtain these numbers.

[topranker]

update...

page 1 on (3rd June 2008) - indexed on 4th June, de-indexed on 5th June <-- still de-indexed
page 2 on (3rd June 2008)- indexed on 4th June, de-indexed on 5th June <-- sill de-indexed
page 3 on (4th June 2008)- indexed on 5th June, de-indexed on 6th June <-- re-indexed 8th June and regained same serp positions.

[g1smd]

Hmm. This is odd. I am looking at a site newly online too.

A site:domain.com search shows 33 www pages. Clicking the omitted results link takes it up to 34 www pages.

A site:www.domain.com search shows 39 www pages. Clicking the omitted results link takes it up to 40 www pages.

A site:domain.com/ search shows 39 www pages. Clicking the omitted results link takes it up to 40 www pages.

The site only has www pages, and all links also point to www URLs too. There is a site-wide non-www to www 301 redirect also in place.

Additionally, there is a bit of variation depending on which Google IP addresses I select to do my search.

[tedster]

I've always assumed it was a way of making sure time-sensitive content appeared immediately

No doubt that Google does do this also - they watch queries for "burstiness" and take special effort with spidering and indexing urls with those terms. But I think that's only part of the picture. Another clue, on AOL site:example.com is usually a lower number of results for me than site:example.com/* on regular Google.

Part of that can be Safe Search - AOL defaults to "strict" whereas Google defaults to "moderate". But that alone is not the whole difference.

these pages are new topics so don't have as many internal links

That's a good observation. It might well be one of the factors involved with the indexed, de-indexed, reindexed cycle.

[majestic12]

Today I checked my test page again, it is now indexed however not even int he top 100,, when it first entered Google it was number 1.

What a strange world this is..

[topranker]

update...
page 1 on (3rd June 2008) - indexed on 4th June, de-indexed on 5th June <-- re-indexed 9th June with bad serp positions
page 2 on (3rd June 2008)- indexed on 4th June, de-indexed on 5th June <-- sill de-indexed
page 3 on (4th June 2008)- indexed on 5th June, de-indexed on 6th June <-- re-indexed 8th June and regained same serp positions.

[wanderingmind]

Getting de-indexed is absolutely normal in my experience. Has always happened, and then the pages come back in 2 days or more. I have seen this happen for 2 years or more...

However. This month has been nuts. Blogsearch pinging not working, sometimes working but pages not making it to the main index for a week.

Since yesterday though, a lot of never indexed and de-indexed pages seem to be on the way back.

[misterjinx]

topranker please could you verify if you are disappearing are your site is around #700 or #900 searching keywords in all different Googles's DCs ?

[topranker]

All seems ok to me with the keywords i've checked, also stats appear to be showing normal activity at around 8,000 to 10,000 a day.

Our site is uk targeted so doesn't do well in .com but ranks pretty high in .co.uk with site links even for generic keywords.

[majestic12]

This is weird.. yesterday my test page was raised to number 8, this morning it is gone again.. surely Google must be having some kind of hiccup?

David

[topranker]

Here is todays update on my three test pages

page 1 on (3rd June 2008)
- indexed on 4th June wirth very good rankings
- de-indexed on 5th June
- re-indexed 9th June with bad rankings
- 11th June ranking well for just one search term, rest poor

page 2 on (3rd June 2008)
- indexed on 4th June with bad rankings
- de-indexed on 5th June
- 11th June still de-indexed

page 3 on (4th June 2008)
- indexed on 5th June with bad rankings
- de-indexed on 6th June
- re-indexed 8th June and regained same bad rankings.
- 11th June now has very good rankings for many search terms

Pages 1 & 2 are relvant to each other so only link to each other.
Page 3 has relevant links from 4 other older indexed relevant pages

All pages are still linked from main page and an (all pages) links page.

[topranker]

Serps still seem very unstable, although our overall site stats are slightly down we have had some reports saying a lot of customer pages have died but still indexed which were previously very busy, so that must mean a also a lot of other pages must have increased in rankings to make up for the loss of traffic on those pages which have been hit.

We really only get to find out about poor performing pages and not pages getting busier.. people only tend to report things if they have a complaint.

It maybe that some pages have been penalized for some reason, but the overall site certainly has not been penalized as its still getting a lot of very good rankings for high traffic keyword terms.

[majestic12]

This morning one thing occured to me regarding my page.. now I use wordpress and am wondering am I guilty of duplicate content?

this is my post page which is fine..

http://www.example.com/category/page-title/

............................................
this is the page that the tag cloud links to

my page is duplicated here?

http://www.example.com/tag/page-title/

................................................

this is the category page.. my page is also displayed here

http://www.example.com/category/page-title/

..........................................................

I checked many popular blogs and to me it seems the same for them.. I am a little confused here?

Regards David

[edited by: tedster at 8:44 am (utc) on June 13, 2008]
[edit reason] switch to example.com [/edit]

[misterjinx]

ok, as I told in my previous post, I am investigating in disappearing positions of many sites and realizes the disappeared pages take positions or near to #700 or near #900 in all DCs.

Then some pages returns in its original ranking positions (or better), instead other pages remain down (also for months).

If checking ranking of your sites you'll find a similar situation, could you please post about it here? Thanks.

[edited by: tedster at 11:55 pm (utc) on June 15, 2008]

[tedster]

misterjinx, have you read about the -950 penalty [webmasterworld.com]? That is what you're describing, and i'm glad tro hear that you do a thorough check. It is important to know whether a url from your site is completely gone from the index, or is still in the index but sent down to the nether regions for a term where it used to rank.

[misterjinx]

Yes tedster and you know (of course) I read it... the url is not completely gone.

A note: in other 2 cases pages completely disappear from index for 2 months (feb-mar '08) to appear again in april 08 ranking around #3-#5. Now lower ranking around #13 but it was for an Italian SEO contest so it is lesser interesting for me now.

[majestic12]

My new site I made the test page for is ranking 1 [Yahoo,Alta vista,Aol] for most terms I target pages for ..

I have about 130 pages now, 20 indexed by Google, and a few for Yahoo....

Right now it looks like if I build a page aimed at "Blue Widgets" I can get to the top of Yahoo but not Google....

Weird thing is..

However... if I "mention" the word Widgets on the Home page the home page gets ranked for the term and not the page that was built for "widgets"....
A while back it seemed if you ranked for something in Yahoo you would have a similar ranking in Google and vice versa, Google UK now though seems to be doing some real crazy stuff... in a way I hope it is broken and that there is a chance they will fix it...

R.e toprankers last post.. my home page is the only pages with links going to it.. [ got some links today for the internals ]

Regards David

David

[johnnie]

<moved from another location>

Hello,

My site was dropped from the index only two days after being crawled and indexed. How can this happen? My site plays by the rules; no cloaking, no spamming, no hidden stuff, no adware, nothing. The only backlinks I have (yet) is from some social bookmarking sites used to get the word out. I do have some affiliate links (it's an aff. site), but I also have several pages of unique, well-written and relevant content.

I don't understand!

[edited by: Robert_Charlton at 11:46 pm (utc) on Aug. 13, 2008]

[johnnie]

It appears I have been re-index and am ranking as good as before. Maybe it's just a matter of different datacenters having to synchronize with some master server?