Welcome to WebmasterWorld Guest from

Message Too Old, No Replies

Googlebot not very active compared to competition..

Loosing freshness?



9:59 am on Mar 23, 2005 (gmt 0)

10+ Year Member

I maintain a large network of websites in alot of different areas so i have quite good insight in crawling activity between different search engines.
I find it strange that googlebot is so passive.
A normal day i get about ten times more bot hits from MSN spider and about five times more bot hits from yahoos spider.
I also see that MSN is much faster including new sites / pages in then google.
Is googlebot crawling "smarter" or are they loosing the freshness of their index..?

Would be nice to hear others experience.


11:24 am on Mar 23, 2005 (gmt 0)

WebmasterWorld Senior Member ciml is a WebmasterWorld Top Contributor of All Time 10+ Year Member

Google fetches well linked URLs much more often than not so well linked URLs.

I see some getting fetched daily, and others not fetched in over a year.


12:42 pm on Mar 23, 2005 (gmt 0)

10+ Year Member

I got some very well linked sites that are getting fetched daily by google but the number of fetched pages compared to especially MSN and also Yahoo are much lower..
so the point is that both msn and yahoo includes more fresh pages faster then google the way i see it..


4:49 pm on Mar 23, 2005 (gmt 0)

10+ Year Member

I always thought, frequency of Googlebot visits depends on PageRank and number of quality inbound links on other high ranked pages. Higher the PR and inbound links greater is the frquency of visits. I am damn sure the WebmasterWorld has a lot more visits and pages get much faster, because of its high PR and quality inbound links. PR5+ sites do usually get visited everyday or atleast every alternate days.


4:52 pm on Mar 23, 2005 (gmt 0)

10+ Year Member

And not just the frequency of visits but the depth of crawl in your site and the number of pages indexed are also connected to PR.


5:06 pm on Mar 23, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member


Google seems remarkably fast respidering my site, usually catching textual changes within a few days at most. And I don't see a serious bandwidth issue from any spiders though I try to turn off some of the more marginal but CPU-expensive features on pages when I detect that the visitor is a spider rather than a human.

However, Google seems to be lamentably (>6 months?) behind on spidering images, which from what I hear is not untypical.




Featured Threads

Hot Threads This Week

Hot Threads This Month