Any way to get Googlebot to visit more often?

[Mark_A]

I am working on a series of sites that for a long time had absolutely no new content. As a result googlebot only seems to visit once a month.

If gbot visits at the start of a month and finds new links to new content it notes the new links (visible in the cache) but does not follow them until it's next visit a further month down the road.

So effectively if I want to monitor the effect of a particular change, I have to wait more than two months. This is proving very frustrating - I am used to sites that are crawled at least weekly.

Is there any way to increase the frequency of gbot's visits?

[tedster]

New backlinks are a valuable factor for increased crawling - that is, increase your real PR.

[Mark_A]

Hi tedster, we have been getting lots of new links from a PR campaign which has resulted in lots of online articles and links but it does not seem to be having any effect on the crawl rate, yet at least.

[petra]

Have you submitted an xml sitemap?

[Mark_A]

petra - Have you submitted an xml sitemap?

Hi petra, no, at the moment I am not using Webmaster Tools at all. I am trying to make my mind up about whether to jump in or not.

[petra]

I would suggest that you do and do submit a sitemap, can't hurt.

[Mark_A]

petra - I would suggest that you do and do submit a sitemap, can't hurt.

But petra, I just don't trust google!

[nomis5]

Mark_A, glad you posted that. It takes Google ages for new pages and changes to get indexed on my sites as well. I've never quantified it like you have but that matches about what I see.

And yes it is frustrating. It makes evaluating the effects of a change almost impossible.

[tedster]

I just don't trust google!

This is the one case where Google is giving you a lot of their data, and all you are giving them is your email address. The minute you verify your website, the reports are populated with all kinds of data they already have. One example: there's no other way to know how many times your URLs get an impression in the SERPs (without being clicked on).

So it's not like Analytics, where you're giving Google all kinds of extra access to your data.

[Mark_A]

So, WRT WMT.. I already have an analytics account which lists and reports on four related sites. So google knows they are related which is obvious from their content, and they are associated with us, which is also obvious from the company email address. That is just where you would expect someone to be involved with the site to reside.

We also have an adwords account which lists three of the four sites and used to spend a lot of money which recently I have cut in half.

So if I register for WMT again with a company email address, what else am I disclosing to G ... ? just that someone with a company email address is interested in the performance of the company websites. That would be normal.

But what I really want is first page natural listings for up to six of my most most wanted key terms. Will WMT help me to achieve that?

[tedster]

The data you get may help you see things from a new angle, and pinpoint new ideas for improved ranking. Just registering the WMT account alone won't help rankings, of course

[Mark_A]

OK so I joined Webmaster Tools. I had a good look around.

GWMT seems to ignore many real incoming links we have.

Surprised at the top keyword Google thinks for the site.
Didn't see that one coming.

Crawl stats show what I know, crawls are way down.

Well will keep on digging through it.

I was persuaded to change our robots text file. can't be bad!

[Mark_A]

After being persuaded by GWMT to change the robots.txt which had been excluding all our PDF content, the crawl rate (pages per day) has shot up. I assume it is assimilating all the new pdf content. Hope that this strengthens our relevance for our keywords. I would be very dissapointed if it does not.

For some reason GWMT does not have hardly any data on our German site. The two UK sites had loads of data all ready but the German site has none.

[almighty monkey]

My dataset for this is only 4 websites, so don't take this as gospel.

But, it seems clients with an adwords account get spidered more often than sites without.

Makes sense - Because it's instant, adwords sites are more open to abuse by the Viagra sellers of the world than sites that rely on natural listings. Therefore, Google will want to keep a closer eye on them, hence more Googlebot.

Its been observed that, despite Google's denials, Adwords does help with natural listings. I believe them when they say adwords doesn't add any direct boost in the natural SERPS, but if the site is being updated regulary, the information from the additional googlebot scans feeds back into the 'freshness' algorithm, giving the site a artificial boost that way.

Also, ignore the GWMT incoming links tool. In fact, never trust Google for any specific information on offsite links. Lies! Lies I tell you! All lies!

[Mark_A]

Hi almighty monkey, I am not sure that sites with adwords accounts get crawled more often. Our UK site has been running adwords for the last 2 years at least and its spidering went way down in the last couple of months.