homepage Welcome to WebmasterWorld Guest from 54.227.182.191
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

    
Number of indexed pages keeps falling
apauto




msg:3777512
 3:30 pm on Oct 31, 2008 (gmt 0)

About 2 weeks ago, when I did the "site:" command for my domain, it would have about 9000 pages listed in Google. A week ago, this dropped to 2200, and every other day, it keeps dropping by about 50-10 pages. Today it was down to 1190.

It's strange because my traffic is only a little lower, maybe 10%.

I know my site has more than 1190 pages, a it's dynamically built, and I double checked that each has a unique title, meta, and content, so there is no problem there.

Google crawls my homepage about every 12 hours.

Should I be worried?

Thanks guys

 

apauto




msg:3777657
 6:24 pm on Oct 31, 2008 (gmt 0)

I just saw this: [webmasterworld.com...]

Could this be related? Let's keep this discussion to those of us who have noticed their site: results fall the last few days and weeks.

bw3ttt




msg:3777685
 7:07 pm on Oct 31, 2008 (gmt 0)

I have been able to stop and reverse page droppage by getting new inbound links.

apauto




msg:3778283
 5:12 pm on Nov 1, 2008 (gmt 0)

Anyone else noticing a lower site: reading?

apauto




msg:3786499
 7:41 pm on Nov 14, 2008 (gmt 0)

It still keeps falling. Traffic hasn't fallen off though... how bizarre. Went from 9000 pages to now only 7xx.

webdude




msg:3786524
 8:34 pm on Nov 14, 2008 (gmt 0)

I have been doing a lot of digging lately because of a site I have that has been losing pages in the index… at least I thought it was losing pages. In the coarse of investigating, I have been finding a lot of discrepancies and have come to the conclusion that though tools and search operators may be helpful, they seem to be far from accurate and do not fully portray what is in the data and returns. What I found fascinating is that while I perceived that I was losing pages in the index, I actually have been increasing position for some relatively hard to get keywords and phrases. In fact, the site in question just went to #6 for widgets. It seems the more I search and investigate, the more glaring the discrepancies.

I was having a lot of problems with the site and duplicate content. It seems there were several ways of getting to the same page (different URLs) and as we know, this can be a bad thing. The site has a forum that has generated 16,000 topics (some of them on multiple pages) so in essence, I am going to estimate that I have around 19,000 pages total on the site. Now at the height of the duplicate problem, when I did a site:mysite.com, I was getting over 80,000 pages returned. WOW! I fixed all the dupe content issues and now each page has one URL and each has a uniquely generated title, description, keywords and of course, the content is different since it is user generated. I used robots.txt to get rid of the duplicated pages and started to watch what would happen. This seemed to have corrected the problem. Pages started going supplemental and dropped, as far as I can tell. But the pendulum seemed to have swung too far! Within the past month, the number of pages returned using site: have been slowly dropping. Now when I do a site:mysite.com, it only shows 4000 pages. Huh? What’s the deal with that? Not only that, when I do a site:mysite.com/*, I only get about 800 pages. So I am confused, of course. But are the missing pages really not there? I conducted about 200 searches for the pages that I thought were missing and found every single one of them, though the searches were fairly specific. So what does this tell me? The site: operator does not work. All of my pages are there, it’s just Google doesn’t want to count them all with this operator. What does this mean? Not sure, but it is what it is. For every page I find missing, I can find in a search. The tool seems to be broken - like a lot of the tools on G.

rjwmotor




msg:3786618
 1:12 am on Nov 15, 2008 (gmt 0)

I am having a similar problem. My site used to have about 1600+ pages indexed and it too started dropping pages at about 50-100 a day. I am at around 680 at last look. Rankings for major KW's remain unchanged but it seems that my refferals from G are down about about 20%. Don't know if this is related or not as at best guess it would appear that I'm losing some of my long tail traffic. My site is also somewhat seasonal and does slow somewhat in the winter time. I recently lost all my old stats so I can't even compare it to previous years.
On another note, orders are WAY DOWN! I've only done about half of my regular business so far this month. I'm hoping this is due more to the economy than anything. I also checked on G Trends and noticed that interest has been gradually falling for these products over the past year so it could be a combination.

I also was told in another thread that AOL has a more accurate count of pages in G's index as they supply the results.

minnapple




msg:3786627
 1:57 am on Nov 15, 2008 (gmt 0)

I started to do some testing to combat the lack of depth indexing.
I am placing {noindex} on pages that have either no se value, [i.e. faq pages etc..] or have at least two other pages that well represent the targetted search phrase.
So far I am seeing positive results, but this is early in the testing phase.

webdude




msg:3786703
 7:41 am on Nov 15, 2008 (gmt 0)

Mmmm... that's intersting. I just did {noindex} on all pages beyond the original... in other words, if a post had more then 1 page, the first gets indexed and the rest don't. No need to clutter up the index with junk that's not needed. Isn't that what it's all about?

;-)

drall




msg:3786793
 2:42 pm on Nov 15, 2008 (gmt 0)

I have made several posts about this over the last month, we have been watching our site: command drop from 80,000 results to 20,000 results now and have actually seen our overall traffic increase.

My best guess is that this is simply a change in how google reports site figures. Our page counts have been dropping by a couple thousand pages per week for a month and continues to drop.

All rankings stable and some increases actually. 10 year old site, PR 7, around 1 million backlinks.

rj87uk




msg:3786822
 4:17 pm on Nov 15, 2008 (gmt 0)

minnapple - Where are you putting the no index tag?

Is that in the robots.txt or on the page?

RJ...

minnapple




msg:3787121
 5:37 am on Nov 16, 2008 (gmt 0)

on the page
I am also considering doing a no follow to certain pages but I would not rather do this because of the error checking involved in doing this.

wchan07




msg:3787124
 5:44 am on Nov 16, 2008 (gmt 0)

hi all, i was worried about the same thing. I think the count is off for some reason too. site:www.mydomain.com had 200,000 in mid october. now it is at 37,000. I freaked.

My site is organized alphabetically a-z, so i did a check on site:www.example.com/z/ for the letter "z". I got 126 results only. It would normally be a few THOUSAND

Then I started doing subdirectories from Z, like site:www.example.com/z/subDir1, site:www.example.com/z/subDir2/... and was pleasantly surprised to see all my pages in there. So the actually pages indexed is much higher than 126. Basically it says "126" but actually has a couple thousand this is strange, i hope they fix this as it freaks me out.

[edited by: tedster at 6:42 am (utc) on Nov. 16, 2008]
[edit reason] switch to example.com - it can never be owned [/edit]

apauto




msg:3787328
 5:33 pm on Nov 16, 2008 (gmt 0)

wchan07 - did you get lower traffic?

My site: pages have been falling, but traffic has remained the same.

rj87uk




msg:3787822
 11:16 am on Nov 17, 2008 (gmt 0)

minnapple, I would like to test this out because I have a lot of "fluff" pages that I don't want to rank or any internal link juice passed onto them.

What is the code?

... I have never done anything like this before..

toonarmy




msg:3787839
 11:57 am on Nov 17, 2008 (gmt 0)

same problem here, 3 months ago i had 70000 pages indexed but every few days i loose a few 1000 more, last night i was down to 17000 and on checking this morn i am now down to 12000, funny thing is we have lost about 50% traffic not all of it

webdude




msg:3787984
 4:14 pm on Nov 17, 2008 (gmt 0)

Hey rj87uk, try this...

<meta name="ROBOTS" content="NOINDEX,NOFOLLOW">

Page will not be indexed nor links followed with this tag.

rj87uk




msg:3788297
 10:29 pm on Nov 17, 2008 (gmt 0)

Thanks webdude! I will give that code a try on some pages that shouldn't be in the index!

RJ

minnapple




msg:3788397
 1:21 am on Nov 18, 2008 (gmt 0)


I did
<meta name="ROBOTS" content="NOINDEX">
Not sure if I want to do the nofollow at this point

ecmedia




msg:3788873
 3:17 pm on Nov 18, 2008 (gmt 0)

Seeing a massive drop in indexed pages with no change in traffic. Looks like a glitch.

latimer




msg:3789080
 7:55 pm on Nov 18, 2008 (gmt 0)

Also seeing massive drop in indexed pages over the past few weeks. did the site:searchmysite.com/foldername suggested by wchan07 and this didn't change the numbers in the index here.

I agree, there doesn't seem to be the type of drop in traffic that would be expected for this many pages being missing in the index.

Noticing that when clicking the cache link on the pages in the index, that many of them don't return a cached version. Is this typical? Seems to be quite a lot of them.

One other observation is that we provide template type content pages for a number of independently owned webstores that use the same folder and file formats attached to different domains These pages have always been prety steady in the index for each domain, although don't rank very well except on the sites that have built some inbound links etc.

The crazy thing is that finding these pages in the index for one of the sites through search site:www.sitenumber1.com/folder1 and then clicking on the cache link and seeing that the cache version of some of the pages is showing from www.sitenumber2.com. The cache link url and the version that the cache link land on showing different domains.

Some pages that sre too similar to get filtered out based on duplicate content within a site, but never saw this type of thing before, and wonder if this is just part of the glitch, or we have to make sure that no two sites use the same /folder1/file1.htm addresses in template driven pages. Any thoughts on this appreciated.

latimer




msg:3789081
 7:55 pm on Nov 18, 2008 (gmt 0)

Also seeing massive drop in indexed pages over the past few weeks. did the site:searchmysite.com/foldername suggested by wchan07 and this didn't change the numbers in the index here.

I agree, there doesn't seem to be the type of drop in traffic that would be expected for this many pages being missing in the index.

Noticing that when clicking the cache link on the pages in the index, that many of them don't return a cached version. Is this typical? Seems to be quite a lot of them.

One other observation is that we provide template type content pages for a number of independently owned webstores that use the same folder and file formats attached to different domains These pages have always been prety steady in the index for each domain, although don't rank very well except on the sites that have built some inbound links etc.

The crazy thing is that finding these pages in the index for one of the sites through search site:www.sitenumber1.com/folder1 and then clicking on the cache link and seeing that the cache version of some of the pages is showing from www.sitenumber2.com. The cache link url and the version that the cache link land on showing different domains.

Some pages that sre too similar to get filtered out based on duplicate content within a site, but never saw this type of thing before, and wonder if this is just part of the glitch, or we have to make sure that no two sites use the same /folder1/file1.htm addresses in template driven pages. Any thoughts on this appreciated.

latimer




msg:3789105
 8:33 pm on Nov 18, 2008 (gmt 0)

a few months ago a google sitemap was added to one of the template stores as a test and did seem to improve the indexing a bit before the recent decline started. prior to that sites have all been spidered naturally pretty deep. This is the only thing that we think might have changed to cause the strange cache entanglement between domains mentioned in previous post. Anyone have similar experience at all like this? probably going to remove the g-sitemap.

experienced




msg:3795366
 11:20 am on Nov 27, 2008 (gmt 0)

my site is actually have all new content written by us only and there are hardly any page which is a duplicate of any other page in the whole site. there is no forum or blog placed where post might be shared or something. it is a site placed under several folders and same on the root diffferent content & different style. Is it only the issue of inbound links..... we usually gets 20- 30 OK OK links a month...

incrediblehelp




msg:3795489
 3:39 pm on Nov 27, 2008 (gmt 0)

Why does it matter? I suspect only as handful of your pages are getting traffic to your site anyways.

anallawalla




msg:3795905
 10:26 am on Nov 28, 2008 (gmt 0)

There is another thread like this where I mentioned last week that we lost exactly 1,000,000 pages out of 2,400,000 indexed pages. Curiously, traffic has been the highest on record, so I think this is a cosmetic glitch. The page count dropped to about 1,200,000 but now it has inched back to 1,600,000.

tedster




msg:3796178
 7:09 pm on Nov 28, 2008 (gmt 0)

I completely agree. If traffic is solid, then those numbers for the site: operator are wrong at the moment.

Google is clearly changing something in the way they store data. That change makes calculating the site: numbers (which are only ever estimates) a problem. We've gone through cycles like this before.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved