Welcome to WebmasterWorld Guest from

Message Too Old, No Replies

Why has googlebot stopped crawling site since april 9th

Carwls every day on our other site.



1:59 pm on Apr 15, 2005 (gmt 0)

10+ Year Member

Hi all and thanks in advance for any help you can offer. We own two sites. One older one which is crawled daily by google and a newer one that was last crawled April 9th. Why would that be? I thought that once google begins to crawl it does so daily. We have added a lot of new content and it isnt being piced up due to no new crawl. What can we do? Thanks


2:23 pm on Apr 15, 2005 (gmt 0)

10+ Year Member

(IMO) Google crawls the site depending upon a few factors, one of which is it's knowledge of how often the site is updated.

During a recent ovehaul of my site (over a week), G just happened to index the site during updates.

I returned the next day and discovered additional updates (due to my updating...)...and crawled the site again..

My best guess it it thought that the site was being updated very frequently during that week, and ended up coming every 20 minutes or so looking for updates..

since the update, it's slacked off to once a day (or so)...however I make small updates every couple of days to keep it fresh, and entice indexing of my site.

This also holds true to web sites that are never updated, I believe G measures time/updates to compute how often to re-index the site. If you've never changed your site in 24 months, don't expect the crawler to be there anytime soon...it has millions of other sites that ARE updated regularly.

If it's a new site, you probably have no control over when, but once you see that it's indexed a page or 2 do a small amount of changing to the pages so that it at least thinks the site is kept fresh. I believe that will help in the long run.


2:32 pm on Apr 15, 2005 (gmt 0)

10+ Year Member

I dont know how accurate your opinion is because our other site is crawled daily and hasnt been updated in approximately 3 years. This site newer site is being updated daily and still not getting crawled.


1:42 pm on Apr 19, 2005 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

put in a "if modified since" header and watch your crawls fall significantly.

Featured Threads

Hot Threads This Week

Hot Threads This Month