Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Large spike in crawl rate, large decrease in sitemap indexing

         

Silvervox325

2:47 pm on Oct 13, 2016 (gmt 0)

10+ Year Member



A site I manage has seen a HUGE spike in crawl rate for almost all of October, but the number of URLs being indexed in the sitemap report has been steadily decreasing over the same period:
[imgur.com...]

The sitemap is showing 0 errors, nothing is blocked by robots.txt. all URLs within the sitemap are valid, and spot-checks have shown that the URLs in the sitemap are still indexed. We've made no major changes to the site during this time, and I can't find a single reason why the sitemap shouldn't be fully indexed - we're a well-established, completely whitehat site that has ranked moderately well in the SERPS for almost a decade.

Is anyone else seeing similar patterns? Is this just a Search Console bug?

goodroi

5:51 pm on Oct 13, 2016 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



Have you changed the design of the website? Did you include a new freshness element to the pages? Did you have an influx from social? Was there any change to your hosting uptime? Did you gain new backlinks? These are just some of the reasons that can impact Google crawl.

I don't think you are seeing a console bug. I would suggest you view the page as googlebot sees your page. You want to make sure there is significant unique & valuable content on each page and that content is accessible to google. To be safe also double check robots.txt and your page headers to make sure you aren't blocking Googlebot

NickMNS

6:21 pm on Oct 13, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



According to your image, it seems you submitted the site on Oct 7. It is normal that there is a spike in crawl rate when you submit new site maps. I assume that the decrease is a result of Google removing pages from the old site maps that are no-longer valid. Once all the old page are removed you will begin to see the index climb, but the climb seems to take longer. I have experienced this exact same situation before.