homepage Welcome to WebmasterWorld Guest from 54.204.94.228
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

    
GWT: "Too Many URLs for Googlebot" - But All Are Disallowed
eileenw




msg:3776117
 6:56 pm on Oct 29, 2008 (gmt 0)

This is driving me a little batty. I keep getting messages in the Google Webmaster Tools about Googlebot finding too many URLs on a site. However, I took action on this message a while ago and robots.txt disallowed all the offending URLs (filter combos, sorts, etc.) to try to fix this error. But I keep getting the message, now citing URLs which are all robots.txt disallowed! What should I do here? Do I need to JavaScript these URLs or switch everything to drop-downs?

 

tedster




msg:3776156
 8:02 pm on Oct 29, 2008 (gmt 0)

Can you easily make a url removal request for this set of urls? I think that would fix whatever short ciruit is going on.

eileenw




msg:3776197
 8:40 pm on Oct 29, 2008 (gmt 0)

Unfortunately, no.

tedster




msg:3776206
 8:48 pm on Oct 29, 2008 (gmt 0)

Well, then it will take longer. But if your robots.txt is accurate, this issue should still get sorted out over time.

If you have a WebmasterTools account, there is a good tool in there to help you verify that your robots.txt is valid and functioning as you planned.

eileenw




msg:3776227
 9:07 pm on Oct 29, 2008 (gmt 0)

Thanks. Also, I've already checked that the robots.txt is valid. Has anyone else with this issue seen this message stop after putting sorts/filters into drop-downs or JavaScript?

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved