homepage Welcome to WebmasterWorld Guest from 54.196.24.103
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld

Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

    
Google WMT Indexing Gone Wild?
Komodo_Tale




msg:4166076
 8:41 pm on Jul 7, 2010 (gmt 0)

Google has gone nuts. It is combining URLs and trying to crawl them. The index wend from 10k to 126k/79k uniques. GWMT is also displaying a warning:

Googlebot found an extremely high number of URLs on your site: http://www._____.com/ July 7, 2010

Googlebot encountered problems while crawling your site http://www._____.com/.

Googlebot encountered extremely large numbers of links on your site. This may indicate a problem with your site's URL structure. Googlebot may unnecessarily be crawling a large number of distinct URLs that point to identical or similar content, or crawling parts of your site that are not intended to be crawled by Googlebot. As a result Googlebot may consume much more bandwidth than necessary, or may be unable to completely index all of the content on your site.


I have checked the site, sitemaps and RSS feed. Everything seems okay.

Has anyone seen something like this?

 

tedster




msg:4166102
 9:21 pm on Jul 7, 2010 (gmt 0)

There are a number of recent posts here about WMT data being off - but none have been about a "too many URLs" problem. That has historically been a very real warning to the site owner.

I would say check out some of these URLs and see how your server responds. Google will crawl not only URLs from your sitemap or your site's internal links. They will naturally crawl URLs that they find in external backlinks, and if an incorrectly configured backlink ends up resolving 200 OK on your server, that can start a cascade of "bad URLs".

Komodo_Tale




msg:4166117
 9:42 pm on Jul 7, 2010 (gmt 0)

I don't know where the links are coming from, but you nailed one thing. They resolve to a custom error page with a 200 server response. That's not good. I was fixated on finding the source I did not look at the server response. #duh

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved