Welcome to WebmasterWorld Guest from 100.25.43.188

Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Google isn't indexing new content

     
8:52 am on Apr 18, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 20, 2004
posts:98
votes: 0


Every day I add new content to my website. It used to got indexed within one week or so. Now it is almost two months ago since new content was indexed for the last time.

I checked my robots.txt for failures and my htaccess but nothing strange. My old content is also still indexed. What can be the cause?

12:25 pm on Apr 18, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 31, 2004
posts:786
votes: 0


Funny you should mention this

I noticed on a couple of PR6 sites that new pages added were not being cashed also.

If the links were off the index page Google would spider them, then would have cashed the page.

If the links were off a site map or another page off the index page (One Deep) google would spider that page but then not follow and cash the new pages.

Having seen this on a number of high ranked sites it looks to me like the new spider is not as effective and this is why some of the Google results are stale

12:27 pm on Apr 18, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 19, 2002
posts:1945
votes: 0


agreed
12:29 pm on Apr 18, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 20, 2004
posts:98
votes: 0


I have a website on which people can create their own site. They get a subdomain and the site is listed on the index for one week or so and then moves to a pr4 overview page.

It really is strange because first the new pages were indexed very soon and now they aren't indexed anymore.

The old pages are still in the index.

12:34 pm on Apr 18, 2006 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member pageoneresults is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Apr 27, 2001
posts:12172
votes: 61


They get a subdomain and the site is listed on the index for one week or so and then moves to a pr4 overview page.

What kind of subdomain?

www.example.com/sub/

or...

sub.example.com/

Every day I add new content to my website. It used to got indexed within one week or so. Now it is almost two months ago since new content was indexed for the last time.

Are you being indexed as frequently as you used to be? How often is Googlebot indexing? MSNBot? Inktomi?

What can be the cause?

Did you change anything in the way you've been adding content? Change hosts? Can you backtrack and determine if there were any changes you made at about the time the indexing subsided?

1:04 pm on Apr 18, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 20, 2004
posts:98
votes: 0


What kind of subdomain?

sub.example.com


Are you being indexed as frequently as you used to be? How often is Googlebot indexing? MSNBot? Inktomi?

Well yes I'm indexed as frequently, because I use Google Sitemaps. I see that they read my sitemaps today.


Did you change anything in the way you've been adding content? Change hosts? Can you backtrack and determine if there were any changes you made at about the time the indexing subsided?

Adding content is the same and the markup of the content also. I did not change hosts. I recently added robots.txt file, to exclude some files... but i checked it 1000 times to see if it was good..
A week or go I deleted the robots.txt.

The robots.txt was meant to exclude some contact/mail pages, because Google replaced the index pages with the contact pages. For now I again don't have a robots.tct file.

1:13 pm on Apr 18, 2006 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member pageoneresults is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Apr 27, 2001
posts:12172
votes: 61


sub.example.com

I do believe there are some recent topics around here on issues with sub-domains and Google's latest algo shifts. I haven't read them in detail as they don't apply to me but you may want to investigate those further.

1:29 pm on Apr 18, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Jan 9, 2005
posts:1094
votes: 2


yep. my new pages do not get indexed, too.
another our site is losing pages.

spring cleaning? ;)

3:23 pm on Apr 18, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 20, 2004
posts:98
votes: 0


Hmm, will it stay like this way, that subdomains won't be indexed anymore? Or is it just a few months like this?
4:23 pm on Apr 18, 2006 (gmt 0)

New User

10+ Year Member

joined:Apr 18, 2006
posts:3
votes: 0


I am having the same exact problem for my site. My site is broken up into a domain with a few subdomains. Each subdomain is its own unique site. I was part of the BD supp hell with this domain and my guess is there are still many problems with this.

On one subdomain alone, the BD spider has the following index stats:

Feb 2006 238158 hits from the BD spider.
Mar 2006 184573 hits from the BD spider.
Apr 2006 196280 hits from the BD spider.

The amount of pages for this site in google as of today:

109 pages.

#*$!? Big Daddy is obviously still having a lot of problems getting content added to the index.

10:49 pm on Apr 18, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 31, 2004
posts:786
votes: 0


This is real odd

A site we work on PR6 has specific pages like:-
/Pink-widgets.html PR5s, with specific relevent dynamic pages off it like Pinky+Yellow+widgets which are PR3.

I notice that these dynamic pages are not in the serps yet the page has been spidered recently. In other cases the links are just not followed and the pages cashed. I cant just restrict this to Dynamic pages, ive seen it on statics also where links off them to other statics are just not cashed. These are for very specific search terms with low results numbers so you would expect them to show somewhere on three / four keyword search requests.

The only conclusion i can come to is that either:-
1. Google is not updating its index with new pages or is simply not adding even updated pages or prepared to change the pages it has already in its BD data centre - hence why some results look stale.

Or

2. Google has imposed some sort of penaulty to this site where its just not going to index some of the sites pages no matter how relevent - But if so, wouldnt any restriction apply to the entire site?

Amyone any ideas?

3:48 am on Apr 19, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:Dec 2, 2002
posts:87
votes: 0


google has really s*c*ed in this aspect lately! as i've said before, it it were not for scraper pages, doorways, 10 million link pages etc.. google's highly inflated indexing stats they boast would not come anywhere near that. both other engines do a much better job of getting the content cached and indexed. here's a good example why. about 6 month's ago, we did a tv co-op where we pitched in 500,000 for air time. we were givin exclusive rights to sell that certain product on the internet. msn and yahoo had us up witihn the first 2 weeks of the infomercial airing, while google has not even cached the pages yet. the site is 6 yrs old, pr6, and highly competitive in our sector. no excuse for it

(btw) adwords never saw a dime of our money, we gave it all to overture =)

4:39 am on Apr 19, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 5, 2003
posts:689
votes: 0


This is not new and is not limited to subdomains.

New pages on many sites are not getting indexed. I have a site which was crawled almost every single day, and new pages would get into the index every week - but after Big daddy, crawls are pathetic - a couple of pages a day, new pages being crawled but not getting into index - and others have complained about the same problem in other threads.

And this varies from site to site. I have a site on which post Big Daddy, pages enter index within 48 hours too.

Earlier, it used to be that if you have decent PR and your site was continuously being updated with new articles, they would be crawled within a week. No such guarantees now.

5:40 am on Apr 19, 2006 (gmt 0)

Senior Member

joined:Oct 27, 2001
posts:10210
votes: 0


I haven't noticed any change for my site. I see the same pattern that I've seen for months (or longer):

- New pages that are linked from my home page or a section's main page get indexed very quickly.

- "Inside" pages of articles (pages 2, 3, etc.) take longer to show up in the index.

5:50 am on Apr 19, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Oct 16, 2003
posts:971
votes: 0


Speculation:

  • Either a new algorithm for selecting crawl target order is in place, or a special operation is still ongoing.
  • If new algorithm, one goal is decreasing unneeded fetches.
  • If new algorithm, it is not nearly as dependent upon PR as the old algorithm (which was clearly dominated by PR).
  • If new algorithm, it's volume is reduced enough that (on my site at least) it's going to take a while to have enough data to analyze what the new algorithm might be.

8:04 am on Apr 19, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 28, 2005
posts:164
votes: 0


I'm having problems getting pages indexed too. I started this thread here but didn't get any insight as to why this was happening. [webmasterworld.com ]

Tested everything via Google's account tools and all is working fine, have a sitemap that is visited regularly etc.

Pages are being visited by the bot - albeit slowly.

All I could think was that the algo had changed and that my site's PR wasn't warranting deeper indexing anymore.

8:45 am on Apr 19, 2006 (gmt 0)

New User

10+ Year Member

joined:Mar 9, 2006
posts:2
votes: 0


Haven't noticed any real difference in my clients sites...they're up and rolling and getting crawled regularly...one site is really weird, suddenly it gets crawled every day, rankings in the SERP's (on different lesser competitive, but highly relevant keywords) got up...following all that, visitors numbers doubled...

sctrange behaviour of one site, but like i said the others haven't really changed..

9:11 am on Apr 19, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member tigger is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:July 16, 2000
posts:3560
votes: 0


yep suffering from the same problem pages not getting indexed, pages that have PR not showing as cached its all acting very odd right now
9:22 am on Apr 19, 2006 (gmt 0)

New User

10+ Year Member

joined:Apr 5, 2006
posts:8
votes: 0


By the way Google does index new content if its worthy.
9:25 am on Apr 19, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member tigger is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:July 16, 2000
posts:3560
votes: 0


the content on my site is worthy! all my own stuff and thats not been indexed despite being up for 4 weeks
9:27 am on Apr 19, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 13, 2004
posts:1425
votes: 0


I very recently had a new page indexed in about 2 days. No magic about it.
I just linked to it from another page that gets crawled regularly. -Larry

Dayo_UK

9:31 am on Apr 19, 2006 (gmt 0)

Inactive Member
Account Expired

 
 


It is clear that on some sites Google has a problem and other it does not.

Sites who are not getting indexed from my obversations are the same ones that had the supplemental issues, are the same ones that have canonical issues etc.

Nothing particullary new - Google still have problems in the same area they have had problems in for a while.

9:33 am on Apr 19, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member tigger is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:July 16, 2000
posts:3560
votes: 0


WONDERFUL!

Dayo_UK

9:41 am on Apr 19, 2006 (gmt 0)

Inactive Member
Account Expired

 
 


>>>WONDERFUL!

I know - it does seem to be the case though.

I would also say sites that have been recently corrected or new at the moment are not getting indexed or re-indexed.

10:17 am on Apr 19, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member tigger is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:July 16, 2000
posts:3560
votes: 0


supopose it really depends on what you class as new! I put a site up 18 months ago and over the last 6 months since its been ranking have been adding more content to it (orginal content) between 4-6 pages a day. But it just seems to be struggling to do deep crawls - where is the old Gbot gone
11:05 am on Apr 19, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:Aug 6, 2004
posts:84
votes: 0


I very recently had a new page indexed in about 2 days. No magic about it.
I just linked to it from another page that gets crawled regularly

Well, what you call magic seems to have run out for the rest of us. Just be happy that your site isn't bugged by big daddy like some of us are.

I would also say sites that have been recently corrected or new at the moment are not getting indexed or re-indexed.

I would have to agree entirely on this. Pages that used to take a day or two to get indexed are now taking weeks. And this is content that is more than worthy with many inbound links.

It just seems that if you had the supp hell problem in the past your still messed up. what is even more annoying is that my site is completely cleaned up from canonical issues and shows that way in the index.

It just seems that something happened behind the scenes during the supp hell fiasco which has not been resolved yet.

11:27 am on Apr 19, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 5, 2003
posts:689
votes: 0


...exactly what is not working for some of us anymore, Larry.

I noticed this on one site, and am beginning to suspect its starting on a second site too. Both sites with decent PR and one would get 500-1000 pages crawled every day, including any new pages linked from homepage.

11:28 am on Apr 19, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 20, 2006
posts:42
votes: 0


Do you think that the time to load a page is important or depends on what google get indexed?
11:36 am on Apr 19, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member tigger is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:July 16, 2000
posts:3560
votes: 0


nope the bulk of my pages are the same size so loading times are almost the same so I don't see that as an issues - what I do see is Gs lack of being able to crawl sites properly! (thats is the ones here that are having a problem)
11:50 am on Apr 19, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 31, 2004
posts:786
votes: 0


Add a new page link to your index page and Google will now take longer to cash it. Any links to pages off it and its a few weeks minimum.

Add a link on an internal page pointing to a new page and it will take even longer.

This is not sandboxed, low PR site stuff pages im making referance to etc etc - This is on High PR7 authority level type sites rich in content that im seeing this problem on.

Either the new google bot cant cope OR Google has decided to slow down the adding of new content by a few weeks as another perthetic attempt to try and push up adwords revenue.

Seriously, imo Google is not the search engine it once was. This new infastructure may be all singing and dancing for the staff at Google but for the end user the basic search that Google were once famous for leading the way with is no more. This BD update has been a nightmare for webmasters and a disaster for search users.

The number of pages missing from the index is simply amazing compared to how it was pre BD yet they CLAIM to have more pages indexed and report higher result numbers.

This 53 message thread spans 2 pages: 53