Forum Moderators: open
Even with daily visits, the only fresh page to appear and stay in their index has been the main page. Others have showed with fresh tags only to be replaced with previous versions the next day. Three pages have appeared and then vanished for the index.
Has anyone else noticed this increase in Googlebot2's activity and have you had the same problem getting fresh pages in Google? Does this increased crawling have any significance in tems of when Google will finally add pages crawled in the past month? Are there any on page factors that would cause Google to crawl, index and display a page in searches and then revert to a previous version?
I've not followed Googlebots crawling patterns long enough to know if this is normal, but this seems to indicate a drastic change. My page rank, search positions, robots file and content are basically unchanged since Brandy, so these are not factors. Any thoughts?
The freshbot will grab a few pages at a time and show them quickly, but a deep crawl gets them into the index more permenantly (though these are kindof the same thing now..)
My sites been up for about 4 months, and as i have been working on backlinks, i have seen a huge increase in activity..
Month one - homepage daily PR NA
Month two - homepage daily + 1 deep crawl PR 3
Month three - 50 pages daily + 1 deep crawl PR 3
Month four - 50 pages daily + 1 deep per week! PR5
(900 pages each deep crawl)
How longs your site been up?
Of course this is only my speculation based on my site and what i've read in these forums...
I've not looked at Googles crawling activity in the past because the pages always ranked high and any updates showed up quickly. It's just since Austin when they dropped off the map that this mattered. After making the changes, it takes forever for them to show up and then they often revert to previous versions.
The increased crawling has continued a 2nd day, so I hope this is an indication they will make it into the database sooner.
Certainly high PR seems to help crawling frequency. Whether this is due to the high PR itself, or simply a higher probability of the bot coming round from your links is hard to say. Probably the latter.
I've seen a lot of worries and concerns about G's recent crawling patterns from webmasters, but (touch wood) my PR 6/7 continues to get almost daily crawls.
(I try to avoid the "I'm all right Jack - so you're obviously just whingers" stance, which is now seen so frequently on these boards. It's called 'mocking the afflicted' as I recall :)
Funny though, my PR 5 'academic' / info site is being badly crawled, and last time I checked showed PR 0 on all its internal pages. It's got very simple linking: essentially just 'home' or 'next page' (in true academic, un-optimised style.) But it seems to be suffering.
It's just a good thing I don't live off it :)
Cheers
I also have another domain with the main page a PR5 with deeper pages PR4 where pages are crawled frequently, show fresh in Google and then revert to previous versions. This one is in the Yahoo Directory, Google's Directory and has remained stable through all of the updates.
It's frustrating to work to improve your content, see your efforts rewarded, then see the new pages drop again. I know my experiences are not unique and Google can't be satisfied with their current stale content. Could it be that Google is broken and in spite of their best efforts, they don't know how to fix it?
So far, this increased crawling doesn't seem to have resulted in any fresh pages hitting the searches.
Does it depend on PR or whether it is the domain's default or index page? Prior to the recent updates I don't remember seeing fresh pages revert to previous versions, only loose their date tag and remain in the results.
Since getting updated pages in the index is more critical than in the past, maybe I just wasn't tracking pages closely enough. Since joining this forum I'm paying more attention to all areas of my site and have picked up some valuable tips. Thanks for the info.
Do you mean a URL cache?
The site has been up for 2 months and the pages that they are spidering (most) are new pages to google but, intil this last update, any pages that googlebot found would be in the index the next "fresh tags"....
teeceo.
They always have, ever since "fresh" dates existed. Nothing new here -- except perhaps in some cases master caches reverted to are older than when Google did clean monthly updates.
