homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

Webmaster Tools "crawl rate" vs. reality

 7:44 pm on May 16, 2013 (gmt 0)

In Webmaster Tools there's a bunch of business about crawl rate and the rate your site is being crawled and access by the Google bots.

It's even got some gizmo where you can turn "down" the crawl rate and seems to put the orders of magnitude in "seconds".

Ok -- that's all fine and good -- but then why is it, if my site is CONSTANTLY being "crawled" by "bots" -- does Google utterly fail to update it in the listings?

My site has some old, putrid, garbage title and description that I changed days ago. If this stupid bot is crawling constantly, why doesn't it change faster?

Something changes on the site, and it's not reflected in the listings for days (sometimes weeks or months) yet the stupid thing is being "crawled" hundreds of times a day?

Again, I'm new to all this, but man, it's frustrating how illogical it all is.



 8:28 pm on May 16, 2013 (gmt 0)

garbage title and description that I changed days ago

Any crawl rate that you assign in WMT is at best a suggestion, and that tool is primarily intended to slow down crawl rates not speed them up. Google will ultimately crawl your site at the rate and frequency it chooses. A few days isn't anything to fret about.


 8:42 pm on May 16, 2013 (gmt 0)

Ok. I don't expect it to be instantaneous, I'm probably just freaking out over nothing. The other sites all eventually changed, though it took forever (a week).

Robert Charlton

 9:34 pm on May 16, 2013 (gmt 0)

littlecubpanda - Simply going through the motions of tweaking titles and getting in the index does not necessarily mean that you will automatically move up in the rankings.

Sounds like you could benefit by some good quality, relevant linking... perhaps suggesting that you also need to promote your content, and that your content has to be worth linking to.

"Freely given editorial links" is what Google is looking for. I would start thinking in timeframes of months, not in terms of days or hours. Crawl rate isn't anything you should be playing with.


 7:04 pm on May 17, 2013 (gmt 0)

While Googlebot may crawl your site constantly, what you need to pay attention to is the crawl rate for those specific URLs that have changed. You may receive thousands of crawls per day across the entire site, but googlebot has to find that specific URL before it can be updated.

I've also sometimes seen googlebot need a couple or a few requests to a page before it makes changes to the web index.

Hang tight, it'll change. If it's really important, you could try giving it more PageRank by linking to it from your homepage if possible. This can speed up the crawl for that URL.


 1:14 am on May 18, 2013 (gmt 0)

Hard to say, considering my site is just a couple of pages. I focus on creating '.com's with just a few pages, since people tend to give such priority to '.com's over blogs, news sites, etc. There isn't much to crawl, in other words, it's pretty straightforward. We'll see. It's been about four days, and my old crap is still in the listings and barely showing life within 100 pages of listings... with the changes I made, it should be in the top ten. It's crazy if it's not put up in the top ten.


 4:55 am on May 18, 2013 (gmt 0)

with the changes I made, it should be in the top ten

I don't see how you get from A to B.


 5:19 am on May 18, 2013 (gmt 0)

You think Google is slow - look at Bing. Bing's crawl rate is not super fast.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved