Welcome to WebmasterWorld Guest from 18.212.222.217

Forum Moderators: Robert Charlton & goodroi

What makes google crawl rate change?

     
1:10 pm on Aug 29, 2018 (gmt 0)

New User

joined:Aug 29, 2018
posts: 1
votes: 0


The pages crawled per day have fallen drastically in a single day. From average of 20-30k per day to 4k in one day. What could be the reason?
3:35 am on Aug 30, 2018 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:12913
votes: 891


One cause might be the reason Googlebot is at your site.

We're in the Mobile-first (re)Index update. This is an ongoing process where all sites are being recrawled and revaluated for this new index.

We also just went through a Core update, where some sites may have seen added activity from Googlebot.

Then there's the normal web spidering where Googlebot follows incoming links from other sites.

You may also see more crawls if your site is getting high traffic.
4:20 am on Aug 30, 2018 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:11566
votes: 182


welcome to WebmasterWorld [webmasterworld.com], dheeraj3600!

if you analyze your server access logs for googlebot requests from a 20-30K day and compare it to a 4k day it may give you some clues.
12:26 pm on Aug 30, 2018 (gmt 0)

Preferred Member

10+ Year Member Top Contributors Of The Month

joined:Feb 5, 2004
posts: 517
votes: 48


Doesn't answer your question directly but it does the title of the topic...

if I update my theme, depending on the amount of changes, this will trigger 3-4x increase in traffic from Googlebot for a few days to a week.
10:57 pm on Aug 30, 2018 (gmt 0)

Preferred Member

10+ Year Member Top Contributors Of The Month

joined:Dec 12, 2004
posts:647
votes: 10

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members