homepage Welcome to WebmasterWorld Guest from 54.166.108.167
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

    
Crawl Rate - Now Active in Google Webmaster Tools
Bewenched




msg:3124610
 8:34 pm on Oct 17, 2006 (gmt 0)

In google webmaster tools you can now see the number of pages crawled, kilobytes downloaded and time spent downloading pages. You can even adjust the crawl rate.

Mine is currently set to "normal" and can adjust it to slow down, however I cannot make it go faster as the button is disabled.

 

tiori




msg:3124622
 8:45 pm on Oct 17, 2006 (gmt 0)

Hey, that's pretty slick!

travelin cat




msg:3124638
 8:51 pm on Oct 17, 2006 (gmt 0)

Bewenched,

On the bottom of the page is this:

The rate at which Googlebot crawls is based on many factors. At this time, crawl rate is not a factor in your site's crawl. If it becomes a factor, the Faster option below will become available.

Ma2T




msg:3124706
 9:27 pm on Oct 17, 2006 (gmt 0)

Very interesting.

On one of my sites I could select "faster" it currently crawls about 800 a day which is enough for me. I won't change it as I don't want the server under any extra strain.

ayalon




msg:3124727
 9:36 pm on Oct 17, 2006 (gmt 0)

Intressting!

I can only select slow or normal.

Any opinions or actual views on when you can select fast?

goubarev




msg:3124755
 9:56 pm on Oct 17, 2006 (gmt 0)

Yeah, pretty cool...
For one of my larger sites I have:

Pages: Max/6310 - Avg/1796
Kb: Max/334995 - Avg/79489 (80Mb per day!)
Mil sec: Max/3063 - Avg/1749

and for this site I do have the option to go "Faster"
I wonder how much faster can it go...
I'll switch it to faster and will keep you guys posted...

TBW - looking at the graph I've had big spike end of Aug and right now...

Ma2T




msg:3124757
 9:57 pm on Oct 17, 2006 (gmt 0)

Makes me wonder how much "faster" is, faster would be nice for sure, im tempted, but then again I don't want google hammering my server too much when it's not really needed.

Bewenched




msg:3124780
 10:24 pm on Oct 17, 2006 (gmt 0)

Yea .. I don't think i'd want it much faster... I can only select normal or slow. We have a very large site.

OutdoorMan




msg:3124790
 10:32 pm on Oct 17, 2006 (gmt 0)

Anybody else seeing this message?:

"We've detected that Googlebot is limiting the rate at which it crawls pages on your site to ensure it doesn't use too much of your server's resources. If your server can handle additional Googlebot traffic, we recommend that you choose Faster below."

I wonder if the choise 'Faster' will have an effect and if it will speed up the proces for me (I'm still waiting to see more pages from my new site -- other than my root page -- in the SERPs).

Bewenched




msg:3124793
 10:34 pm on Oct 17, 2006 (gmt 0)

My only concern about selecting faster (If I had that choice) is if google finds too many new pages too fast that it will throw some sort of spam filter on the site.

OutdoorMan




msg:3124798
 10:41 pm on Oct 17, 2006 (gmt 0)

My only concern about selecting faster (If I had that choice) is if google finds too many new pages too fast that it will throw some sort of spam filter on the site.

For the time beeing my website only have about 60 pages that can be indexed.

[edited by: OutdoorMan at 10:42 pm (utc) on Oct. 17, 2006]

Whitey




msg:3124864
 12:00 am on Oct 18, 2006 (gmt 0)


System: The following message was spliced on to this thread from: http://www.webmasterworld.com/google/3124862.htm [webmasterworld.com] by tedster - 3:47 am on Oct. 18, 2006 (EDT -4)


Absolutely fantastic guys - congratulations on introducing this crawl rate control into WebmasterCentral.

[googlewebmastercentral.blogspot.com...]

How will this folks out there on WebmasterWorld?

I wonder how quickly will it kick into action and if this mean pages will get cached quicker on all DC's?

Now all i ask is for G to identify the problems that cause a site to be filtered, so that webmasters can see if they can fix or send a reinclusion request legitimately [ but at least this is a big step ]

[edited by: tedster at 7:34 am (utc) on Oct. 18, 2006]

eriky




msg:3127048
 3:02 pm on Oct 19, 2006 (gmt 0)

I've just set it to go faster. This option has been there all week for me by the way but now I'm also getting some nice graphs. I have on average 15K pages indexed daily. I never realized G was so active on my site.

netmeg




msg:3127079
 3:24 pm on Oct 19, 2006 (gmt 0)

In my account, all of the sites where the 'faster crawl' option was offered are database driven, and I'm a little wary of enabling them all for faster crawling until I see how much impact it's going to have on database queries. So I'm just taking one site at a time, starting with the smallest (one of my personal sites)

That said, all the new tools are pretty cool; I appreciate all the improvements they keep making to Webmaster Central. Now, if they'd just make it so that I don't have to scroll through four pages to see all my sites...

eriky




msg:3127158
 4:26 pm on Oct 19, 2006 (gmt 0)

Why don't you use caching? (e.g. smarty for php sites)
I always use caching for my sites to offload my server.

TechnologyGuide




msg:3127555
 9:17 pm on Oct 19, 2006 (gmt 0)

This is pretty sweet. We have the faster option available. Oddly it only engages for 3 months then will revert to normal. I wonder if it will remind me to approve the faster speed.

panicbutton




msg:3127815
 1:00 am on Oct 20, 2006 (gmt 0)

LOL, this "Crawl Rate selector" reminds me of the "TURBO" button on my old computer.

Hmm, hang on, that button didn't actually do anything.

But hey, the placebo effect works for me!

Maybe we could also get some "go-fast" stripes for Webmaster Tools as well!

etgsgroup




msg:3127875
 2:12 am on Oct 20, 2006 (gmt 0)

Very interesting.
Nice services.

Idris




msg:3128172
 9:02 am on Oct 20, 2006 (gmt 0)

So what would an ideal graph look like for a growing site - continuing in an upward trend, a straight line or general fluctuations (and could you presume that sites with graphs which don't match this trend will be suffering in Google SERPS)?

[edited by: Idris at 9:02 am (utc) on Oct. 20, 2006]

Frank_Rizzo




msg:3128209
 9:39 am on Oct 20, 2006 (gmt 0)

What would be good is to set a time window for G to visit. Here in UK maybe we could choose 1am to 6am GMT etc.

That would be a cool feature as G could then run at fast speed for all sites and we wouldn't have to worry about server hammering.

Jordo needs a drink




msg:3128655
 3:21 pm on Oct 20, 2006 (gmt 0)

The graphs are great...

I actually identified a problem (my bad coding) with a site of mine that I suspected, but didn't realize how bad it was until I saw that since the beginning of Oct(when I made the code change), the number of pages Googlebot was crawling daily drastically dropped, while at the same time, the time it was taking the bot to download the pages, drastically increased.

netmeg




msg:3128662
 3:23 pm on Oct 20, 2006 (gmt 0)

Why don't you use caching? (e.g. smarty for php sites)
I always use caching for my sites to offload my server.

We do for some sites, but it's not practical for everything.

Jordo needs a drink




msg:3135646
 3:13 pm on Oct 26, 2006 (gmt 0)

Has anyone narrowed down how often the crawl stats page updates? It seemed to update for me last Wed and Thurs, but hasn't updated since.

roots




msg:3148585
 12:39 pm on Nov 7, 2006 (gmt 0)

I changed the speed of crawling to "faster" two weeks ago, but I don't see any difference?!

Anyone noticed any improvements after changing it?

OutdoorMan




msg:3148633
 1:41 pm on Nov 7, 2006 (gmt 0)

Anyone noticed any improvements after changing it?

I have a small (in pages, but heavy on content) website, probably smaller than the rest of you guys' websites -- 60 pages in all.

Before the choise of changing crawl rate was available, I only had 13 pages indexed. Now I have 58 pages in G's index.

Maybe 58 pages is a small step for mankind, but it's a huge step for me ;)

roots




msg:3148687
 2:30 pm on Nov 7, 2006 (gmt 0)

and you can see the change on number of pages crawled per day?

OutdoorMan




msg:3148725
 2:53 pm on Nov 7, 2006 (gmt 0)

and you can see the change on number of pages crawled per day?

I can see a change in how often Googlebot crawls my site:

October (31 days - the last 13 days october are with the "faster" option toggled on):
Googlebot 384+31 4.02 MB

(I believe that approx. 50-60% of the 384 hits came after I've changed the crawl rate to "Faster".)

November (7 days so far):
Googlebot 122+3 1.45 MB

That's the statistics I have (AW Stats).

[edited by: OutdoorMan at 2:59 pm (utc) on Nov. 7, 2006]

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved