Welcome to WebmasterWorld Guest from 54.227.5.198

Message Too Old, No Replies

Google throttling crawl even though set to "Faster"

     

burtonator

6:30 am on Nov 15, 2006 (gmt 0)

5+ Year Member



In Webmaster tools I've set my crawl rate to "faster" but it shows this message:

"We've detected that Googlebot is limiting the rate at which it crawls pages on your site to ensure it doesn't use too much of your server's resources. If your server can handle additional Googlebot traffic, we recommend that you choose Faster below."

Is this just a generic message or could googlebot think that some of my pages are taking long to load so it's slowing down the index rate?

In the last 30 days I've really noticed a drop in my referrer traffic with no spikes so I'm worried that Google is holding back our index rate even though I've set it to "Faster"

Kevin

vanessafox

3:21 pm on Nov 15, 2006 (gmt 0)

5+ Year Member



Hi Kevin,

That is the generic message that explains why the faster option is available to you. I see that it is somewhat confusing to still be there *after* you have chosen faster. I'll work on making that less confusing.

burtonator

11:02 pm on Nov 15, 2006 (gmt 0)

5+ Year Member



Hey Vanessa.

Yeah...... it *does* seem confusing. I would just change the wording. The "faster" option had been set for a while so I was thinking the wording meant that sinc emy page load times for lots of random fetches were high (long story) that Google was throttling me even though I had sent it to Faster.

Thanks!

Kevin

nippi

11:09 pm on Nov 15, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



um, i don;t think this is the generic message. I think you;ve set it to faster, google reckons your server can;t handle it, so its slowed down.

If you now choose to drop back down to normal, I doubt that the faster option will be available to you

goubarev

12:16 am on Nov 16, 2006 (gmt 0)

5+ Year Member



Ah, good timing!
I was just getting the results from the "experiment" we've started in this thread:
[webmasterworld.com...]
Right after crawl rate showed up in the WM tools...

back then (Oct 17th 06) I've had these stats:


Pages: Max/6310 - Avg/1796
Kb: Max/334995 - Avg/79489 (80Mb per day!)
Mil sec: Max/3063 - Avg/1749

After setting it to "faster" - today (Nov 15th 06), I have these:
Pages: Max/6310 - Avg/2524
Kb: Max/334995 - Avg/101801 (100Mb per day!)
Mil sec: Max/3421 - Avg/1794

So in general per day I've got Google to spyder 728 MORE pages per day! (... or it seems that way)

Looking at the graph I notice that google spydering slowed down considerably from mid-September to mid-October, which could count towards lower average number on my original post (the tracking starts from mid-August) - so the spydering from mid-Aug to mid-Sept is about the same as from mid-Oct to mid-Nov...

Conclusion: I still don't know if setting it to faster had actually made it faster... it certainly seems to be the case when comparing the last 2 months... but it's only as fast as it was 4 months ago...

AndyA

12:32 am on Nov 16, 2006 (gmt 0)

10+ Year Member



My minimum number of pages crawled has actually dropped slightly since I increased the crawl speed to "Faster".

goubarev

12:38 am on Nov 16, 2006 (gmt 0)

5+ Year Member



In my opinion the minimum number doesn't mean much... since we had a couple of outages - and I'm sure Google couldn't possibly maintain the sypder crawling non-stop all the time either... so it's kind of moot point...
 

Featured Threads

Hot Threads This Week

Hot Threads This Month