Welcome to WebmasterWorld Guest from 22.214.171.124
Forum Moderators: open
The MediaBot crawls deeper into the site without issue. The site runs AdSense.
Could there be anything in the server config that is causing this? It isn't robots.txt. The index page is lo-fi and xenu crawls it fine, as does the searchengineworld sim spider.
Googlebot comes calling, but eats nothing, because it is till trying to digest that extra billion plus pages. What happens to you when you stuff more in your mouth than you can swallow.
Googlebot gobbled, until it had to take complete datacenters away from serving up SERPS, in an attempt to digest the data.
Florida, Brandy, The Non-Update, they were not updates, just indigestion, so those that think the directory ridden serps you are analyzing are cr@p, you are wrong, it came from the other end.
Just speculating like everyone else!
Forum3 is premoderated for a good reason; it used to be a total zoo during updates, with countless overlapping new threads being started. After the change happened, forum3 became much more informative.
ON TOPIC: Sorry to hear that many sites aren't getting crawled much, but ours is getting crawled and freshtagged better than ever. For example, I put two new pages up on Mar 8, and they were in the serps the next day with Mar 9 freshtags. In fact, all of the 30 or so pages that went online in the last few weeks were crawled within a couple of days and in the serps, to stay, soon after. Our index is a PR6... I don't know if that is a factor.
Is it just my sites, or did Google awaken from its deep sleep?
BTW, my experience with Google spotting new pages to crawl and indexing pages of one of my sites - a few hundred-pages of content and mostly added within last 2 months - is very positive. All in done within a couple of days and they all show up in the serps almost immediately. (It's a PR5 site with most likely all other pages PR4 or lower.)
It is not refreshing my PR3 or below pages as much as I would like it to do but am not complaining expecially after realizing that Y! hasn't even crawled most of the pages even once. MSN was in the same situation as Y! but for some mysterious reasons or coincidence, just a couple of hours after I had submitted my site to Y! free addurl service announced a few days earlier, it showed up on my site grabbing all the pages.
All my sites are informational ones and that could be one of the reasons.
How many of you have your sites all linked together in an effort to get google to find you
My sites are not all crazy linked together in any un-natural manner. I considered that perhaps I was getting penalized for having multiple sites on the same IP. The older sites are still getting spidered/indexed fine.
Does anyone think there is a penalty for having a site and links to it magically appear the same day?
Also, the index page went into the index and had a cache etc 2 days ago, but now it's not in the index. Very strange. Seems like its gonna take some time to get into Google these days.