Forum Moderators: Robert Charlton & goodroi
I checked my robots.txt for failures and my htaccess but nothing strange. My old content is also still indexed. What can be the cause?
I noticed on a couple of PR6 sites that new pages added were not being cashed also.
If the links were off the index page Google would spider them, then would have cashed the page.
If the links were off a site map or another page off the index page (One Deep) google would spider that page but then not follow and cash the new pages.
Having seen this on a number of high ranked sites it looks to me like the new spider is not as effective and this is why some of the Google results are stale
It really is strange because first the new pages were indexed very soon and now they aren't indexed anymore.
The old pages are still in the index.
They get a subdomain and the site is listed on the index for one week or so and then moves to a pr4 overview page.
What kind of subdomain?
www.example.com/sub/
or...
sub.example.com/
Every day I add new content to my website. It used to got indexed within one week or so. Now it is almost two months ago since new content was indexed for the last time.
Are you being indexed as frequently as you used to be? How often is Googlebot indexing? MSNBot? Inktomi?
What can be the cause?
Did you change anything in the way you've been adding content? Change hosts? Can you backtrack and determine if there were any changes you made at about the time the indexing subsided?
What kind of subdomain?
sub.example.com
Are you being indexed as frequently as you used to be? How often is Googlebot indexing? MSNBot? Inktomi?
Did you change anything in the way you've been adding content? Change hosts? Can you backtrack and determine if there were any changes you made at about the time the indexing subsided?
The robots.txt was meant to exclude some contact/mail pages, because Google replaced the index pages with the contact pages. For now I again don't have a robots.tct file.
sub.example.com
I do believe there are some recent topics around here on issues with sub-domains and Google's latest algo shifts. I haven't read them in detail as they don't apply to me but you may want to investigate those further.
spring cleaning? ;)
On one subdomain alone, the BD spider has the following index stats:
Feb 2006 238158 hits from the BD spider.
Mar 2006 184573 hits from the BD spider.
Apr 2006 196280 hits from the BD spider.
The amount of pages for this site in google as of today:
109 pages.
#*$!? Big Daddy is obviously still having a lot of problems getting content added to the index.
A site we work on PR6 has specific pages like:-
/Pink-widgets.html PR5s, with specific relevent dynamic pages off it like Pinky+Yellow+widgets which are PR3.
I notice that these dynamic pages are not in the serps yet the page has been spidered recently. In other cases the links are just not followed and the pages cashed. I cant just restrict this to Dynamic pages, ive seen it on statics also where links off them to other statics are just not cashed. These are for very specific search terms with low results numbers so you would expect them to show somewhere on three / four keyword search requests.
The only conclusion i can come to is that either:-
1. Google is not updating its index with new pages or is simply not adding even updated pages or prepared to change the pages it has already in its BD data centre - hence why some results look stale.
Or
2. Google has imposed some sort of penaulty to this site where its just not going to index some of the sites pages no matter how relevent - But if so, wouldnt any restriction apply to the entire site?
Amyone any ideas?
(btw) adwords never saw a dime of our money, we gave it all to overture =)
New pages on many sites are not getting indexed. I have a site which was crawled almost every single day, and new pages would get into the index every week - but after Big daddy, crawls are pathetic - a couple of pages a day, new pages being crawled but not getting into index - and others have complained about the same problem in other threads.
And this varies from site to site. I have a site on which post Big Daddy, pages enter index within 48 hours too.
Earlier, it used to be that if you have decent PR and your site was continuously being updated with new articles, they would be crawled within a week. No such guarantees now.
- New pages that are linked from my home page or a section's main page get indexed very quickly.
- "Inside" pages of articles (pages 2, 3, etc.) take longer to show up in the index.
Tested everything via Google's account tools and all is working fine, have a sitemap that is visited regularly etc.
Pages are being visited by the bot - albeit slowly.
All I could think was that the algo had changed and that my site's PR wasn't warranting deeper indexing anymore.
sctrange behaviour of one site, but like i said the others haven't really changed..
Sites who are not getting indexed from my obversations are the same ones that had the supplemental issues, are the same ones that have canonical issues etc.
Nothing particullary new - Google still have problems in the same area they have had problems in for a while.
I know - it does seem to be the case though.
I would also say sites that have been recently corrected or new at the moment are not getting indexed or re-indexed.
I very recently had a new page indexed in about 2 days. No magic about it.
I just linked to it from another page that gets crawled regularly
Well, what you call magic seems to have run out for the rest of us. Just be happy that your site isn't bugged by big daddy like some of us are.
I would also say sites that have been recently corrected or new at the moment are not getting indexed or re-indexed.
I would have to agree entirely on this. Pages that used to take a day or two to get indexed are now taking weeks. And this is content that is more than worthy with many inbound links.
It just seems that if you had the supp hell problem in the past your still messed up. what is even more annoying is that my site is completely cleaned up from canonical issues and shows that way in the index.
It just seems that something happened behind the scenes during the supp hell fiasco which has not been resolved yet.
I noticed this on one site, and am beginning to suspect its starting on a second site too. Both sites with decent PR and one would get 500-1000 pages crawled every day, including any new pages linked from homepage.
Add a link on an internal page pointing to a new page and it will take even longer.
This is not sandboxed, low PR site stuff pages im making referance to etc etc - This is on High PR7 authority level type sites rich in content that im seeing this problem on.
Either the new google bot cant cope OR Google has decided to slow down the adding of new content by a few weeks as another perthetic attempt to try and push up adwords revenue.
Seriously, imo Google is not the search engine it once was. This new infastructure may be all singing and dancing for the staff at Google but for the end user the basic search that Google were once famous for leading the way with is no more. This BD update has been a nightmare for webmasters and a disaster for search users.
The number of pages missing from the index is simply amazing compared to how it was pre BD yet they CLAIM to have more pages indexed and report higher result numbers.
