Forum Moderators: Robert Charlton & goodroi
Since the beginning Google would grab the files once a week, usually on Fridays. For the past 10 days they have been grabbing it about 7-10 times per day. These files are substantial in size (split across 3 separate XML files) and require some pretty hefty server resources to generate.
Is this happening across the board or is this unique to me? Any possible explanations?
I'm still not sure how this is going to affect the bot activity in the long run, and now much of the sites it peeks at each time, but so far were getting MUCH more regular crawling, with less content perused.
As with everthing from G we'll watch it, make postulations, try to second guess it, then fall flat on our faces.... yey!
*edit* - sorry, just looked a bit more closely. One site is getting it every 3-4 hours, the other 2 are getting grabbed a couple of times a day.
I tagged a few of the pages I know are already listed with a small query string but this isn't showing up in the results pages yet.