homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

Google just can't get it up
Why Google fails where AskJeeves succeeds

5+ Year Member

Msg#: 32683 posted 9:20 am on Jan 6, 2006 (gmt 0)

I run a website with about 4 million pages <snip>. Although the Google spiders are very active and have been pulling 100,000+ pages per day for the last 3 months, few pages show up on Google. See <snip>. Google indexing of this site essentially collapsed in January 2005 when the number of pages was increased from about 1 million to 4 million.

AskJeeves, on the other hand, indexes 95% of the site.

My current working hypothesis as to why these pages don't show up on Google centers on Google's repetitive pulling of pages to test stability and refresh its indexes. Suppose Google has to be able to pull the same page twice over a two week period before it posts to the index. Suppose also that Google has a maximum pull rate per site. Also, suppose that Google expires pages after a month. With more than 4 million pages, Google cannot do repeat pulls fast enough to keep the pulled pages in the index.

Does this make sense to anyone intimately familiar with Google indexing? If this hypothesis is correct, is there a way to get Google to ease the repeatability requirements?

[edited by: engine at 4:40 pm (utc) on Jan. 6, 2006]
[edit reason] See TOS [webmasterworld.com] [/edit]



5+ Year Member

Msg#: 32683 posted 3:19 pm on Jan 7, 2006 (gmt 0)

To test this, I have throttled Googlebot back to about 1/3 of the 4 million pages, even though Google's indexing of Chinese content is starting to rise above the baseline.

[edited by: engine at 5:24 pm (utc) on Jan. 7, 2006]

Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved