Welcome to WebmasterWorld Guest from 54.167.83.224

Message Too Old, No Replies

Any way to get Googlebot to visit more often?

     
7:37 am on Oct 6, 2010 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 15, 2001
posts:1436
votes: 0


I am working on a series of sites that for a long time had absolutely no new content. As a result googlebot only seems to visit once a month.

If gbot visits at the start of a month and finds new links to new content it notes the new links (visible in the cache) but does not follow them until it's next visit a further month down the road.

So effectively if I want to monitor the effect of a particular change, I have to wait more than two months. This is proving very frustrating - I am used to sites that are crawled at least weekly.

Is there any way to increase the frequency of gbot's visits?
1:40 pm on Oct 6, 2010 (gmt 0)

Senior Member

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:May 26, 2000
posts:37301
votes: 0


New backlinks are a valuable factor for increased crawling - that is, increase your real PR.
2:15 pm on Oct 6, 2010 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 15, 2001
posts:1436
votes: 0


Hi tedster, we have been getting lots of new links from a PR campaign which has resulted in lots of online articles and links but it does not seem to be having any effect on the crawl rate, yet at least.
3:47 pm on Oct 6, 2010 (gmt 0)

Preferred Member

10+ Year Member

joined:Aug 26, 2003
posts:568
votes: 0


Have you submitted an xml sitemap?
7:31 am on Oct 7, 2010 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 15, 2001
posts:1436
votes: 0


petra - Have you submitted an xml sitemap?


Hi petra, no, at the moment I am not using Webmaster Tools at all. I am trying to make my mind up about whether to jump in or not.
12:03 pm on Oct 7, 2010 (gmt 0)

Preferred Member

10+ Year Member

joined:Aug 26, 2003
posts:568
votes: 0


I would suggest that you do and do submit a sitemap, can't hurt.
3:22 pm on Oct 7, 2010 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 15, 2001
posts:1436
votes: 0


petra - I would suggest that you do and do submit a sitemap, can't hurt.


But petra, I just don't trust google!
3:38 pm on Oct 7, 2010 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 29, 2005
posts:1868
votes: 41


Mark_A, glad you posted that. It takes Google ages for new pages and changes to get indexed on my sites as well. I've never quantified it like you have but that matches about what I see.

And yes it is frustrating. It makes evaluating the effects of a change almost impossible.
1:47 am on Oct 8, 2010 (gmt 0)

Senior Member

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:May 26, 2000
posts:37301
votes: 0


I just don't trust google!

This is the one case where Google is giving you a lot of their data, and all you are giving them is your email address. The minute you verify your website, the reports are populated with all kinds of data they already have. One example: there's no other way to know how many times your URLs get an impression in the SERPs (without being clicked on).

So it's not like Analytics, where you're giving Google all kinds of extra access to your data.
2:03 pm on Oct 8, 2010 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 15, 2001
posts:1436
votes: 0


So, WRT WMT.. I already have an analytics account which lists and reports on four related sites. So google knows they are related which is obvious from their content, and they are associated with us, which is also obvious from the company email address. That is just where you would expect someone to be involved with the site to reside.

We also have an adwords account which lists three of the four sites and used to spend a lot of money which recently I have cut in half.

So if I register for WMT again with a company email address, what else am I disclosing to G ... ? just that someone with a company email address is interested in the performance of the company websites. That would be normal.

But what I really want is first page natural listings for up to six of my most most wanted key terms. Will WMT help me to achieve that?
2:07 pm on Oct 8, 2010 (gmt 0)

Senior Member

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:May 26, 2000
posts:37301
votes: 0


The data you get may help you see things from a new angle, and pinpoint new ideas for improved ranking. Just registering the WMT account alone won't help rankings, of course
10:59 am on Oct 11, 2010 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 15, 2001
posts:1436
votes: 0


OK so I joined Webmaster Tools. I had a good look around.

GWMT seems to ignore many real incoming links we have.

Surprised at the top keyword Google thinks for the site.
Didn't see that one coming.

Crawl stats show what I know, crawls are way down.

Well will keep on digging through it.

I was persuaded to change our robots text file. can't be bad!
12:36 pm on Oct 13, 2010 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 15, 2001
posts:1436
votes: 0


After being persuaded by GWMT to change the robots.txt which had been excluding all our PDF content, the crawl rate (pages per day) has shot up. I assume it is assimilating all the new pdf content. Hope that this strengthens our relevance for our keywords. I would be very dissapointed if it does not.

For some reason GWMT does not have hardly any data on our German site. The two UK sites had loads of data all ready but the German site has none.
12:50 pm on Oct 13, 2010 (gmt 0)

Junior Member

5+ Year Member

joined:Nov 26, 2009
posts:90
votes: 0


My dataset for this is only 4 websites, so don't take this as gospel.

But, it seems clients with an adwords account get spidered more often than sites without.

Makes sense - Because it's instant, adwords sites are more open to abuse by the Viagra sellers of the world than sites that rely on natural listings. Therefore, Google will want to keep a closer eye on them, hence more Googlebot.

Its been observed that, despite Google's denials, Adwords does help with natural listings. I believe them when they say adwords doesn't add any direct boost in the natural SERPS, but if the site is being updated regulary, the information from the additional googlebot scans feeds back into the 'freshness' algorithm, giving the site a artificial boost that way.

Also, ignore the GWMT incoming links tool. In fact, never trust Google for any specific information on offsite links. Lies! Lies I tell you! All lies!
12:58 pm on Oct 13, 2010 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 15, 2001
posts:1436
votes: 0


Hi almighty monkey, I am not sure that sites with adwords accounts get crawled more often. Our UK site has been running adwords for the last 2 years at least and its spidering went way down in the last couple of months.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members