homepage Welcome to WebmasterWorld Guest from 54.166.96.101
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

    
Google throttling crawl even though set to "Faster"
burtonator

5+ Year Member



 
Msg#: 3156959 posted 6:30 am on Nov 15, 2006 (gmt 0)

In Webmaster tools I've set my crawl rate to "faster" but it shows this message:

"We've detected that Googlebot is limiting the rate at which it crawls pages on your site to ensure it doesn't use too much of your server's resources. If your server can handle additional Googlebot traffic, we recommend that you choose Faster below."

Is this just a generic message or could googlebot think that some of my pages are taking long to load so it's slowing down the index rate?

In the last 30 days I've really noticed a drop in my referrer traffic with no spikes so I'm worried that Google is holding back our index rate even though I've set it to "Faster"

Kevin

 

vanessafox

5+ Year Member



 
Msg#: 3156959 posted 3:21 pm on Nov 15, 2006 (gmt 0)

Hi Kevin,

That is the generic message that explains why the faster option is available to you. I see that it is somewhat confusing to still be there *after* you have chosen faster. I'll work on making that less confusing.

burtonator

5+ Year Member



 
Msg#: 3156959 posted 11:02 pm on Nov 15, 2006 (gmt 0)

Hey Vanessa.

Yeah...... it *does* seem confusing. I would just change the wording. The "faster" option had been set for a while so I was thinking the wording meant that sinc emy page load times for lots of random fetches were high (long story) that Google was throttling me even though I had sent it to Faster.

Thanks!

Kevin

nippi

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3156959 posted 11:09 pm on Nov 15, 2006 (gmt 0)

um, i don;t think this is the generic message. I think you;ve set it to faster, google reckons your server can;t handle it, so its slowed down.

If you now choose to drop back down to normal, I doubt that the faster option will be available to you

goubarev

5+ Year Member



 
Msg#: 3156959 posted 12:16 am on Nov 16, 2006 (gmt 0)

Ah, good timing!
I was just getting the results from the "experiment" we've started in this thread:
[webmasterworld.com...]
Right after crawl rate showed up in the WM tools...

back then (Oct 17th 06) I've had these stats:

Pages: Max/6310 - Avg/1796
Kb: Max/334995 - Avg/79489 (80Mb per day!)
Mil sec: Max/3063 - Avg/1749

After setting it to "faster" - today (Nov 15th 06), I have these:
Pages: Max/6310 - Avg/2524
Kb: Max/334995 - Avg/101801 (100Mb per day!)
Mil sec: Max/3421 - Avg/1794

So in general per day I've got Google to spyder 728 MORE pages per day! (... or it seems that way)

Looking at the graph I notice that google spydering slowed down considerably from mid-September to mid-October, which could count towards lower average number on my original post (the tracking starts from mid-August) - so the spydering from mid-Aug to mid-Sept is about the same as from mid-Oct to mid-Nov...

Conclusion: I still don't know if setting it to faster had actually made it faster... it certainly seems to be the case when comparing the last 2 months... but it's only as fast as it was 4 months ago...

AndyA

5+ Year Member



 
Msg#: 3156959 posted 12:32 am on Nov 16, 2006 (gmt 0)

My minimum number of pages crawled has actually dropped slightly since I increased the crawl speed to "Faster".

goubarev

5+ Year Member



 
Msg#: 3156959 posted 12:38 am on Nov 16, 2006 (gmt 0)

In my opinion the minimum number doesn't mean much... since we had a couple of outages - and I'm sure Google couldn't possibly maintain the sypder crawling non-stop all the time either... so it's kind of moot point...

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved