Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Google only crawling home page since 15 June

And ignoring sitemap.xml information

         

Phil_Payne

4:39 pm on Jun 20, 2006 (gmt 0)

10+ Year Member



Sometime between 19:35 and 20:30 on 15 June the Googlebot stopped an orderly programme of spidering various pages on the site and started spidering the home page exclusively.

I've changed a number of pages and updated the sitemap.xml lastmod before uploading the sitemap and the changed page. I've got a sneaky suspicion that the number of downloads of index.html between successive recognitions of a changed sitemap.xml is equal to the number of changed pages.

Is it possible that there's a bug in sitemaps - the change is not being passed into the GET that the Googlebot builds and the default page - the home page - gets downloaded again.

Has anyone seen this sudden switch to persistent downloading of a home page?

Phil_Payne

5:20 pm on Jun 20, 2006 (gmt 0)

10+ Year Member



It's been pointed out in Another Place that the sitemap has "NEVER" as the changefreq for almost all files. That's not been a problem so far - perhaps Google has now started paying attention to changefreq. I would have thought a lastmod would override that, but it's easy to test - more later.