Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Drop in WMT "Pages crawled per day" - traffic steady

         

virtualreality

12:23 am on Feb 22, 2016 (gmt 0)

10+ Year Member Top Contributors Of The Month



Traffic is steady, the site is up, no server issues but since a few weeks I noticed a steady decline in my pages crawled per day. Is this a reason to be worried and how can I figure out why ? This is a big site 1000+ pages. I do minor updates on the site from time to time so all the information is current.

I have another smaller site and it hasn't done updates for ages and the stats are the opposite there.

JS_Harris

6:19 pm on Feb 23, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Completely normal. Google checks pages frequently at first and assuming they don't change much over time they visit each page less frequently. If, on the other hand, you're constantly changing the content Google will likely come back more frequently. Neither scenario is something to worry about.

lucy24

7:16 pm on Feb 23, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



since a few weeks I noticed a steady decline in my pages crawled per day

Could they be temporarily reallocating their crawl budget? In the last couple of weeks I'm seeing a great increase in "entrapment crawls" like
/directory/ (real URL)
paired with the two redirected forms
/directory
/directory/index.html
and also requests for certain URLs that have been individually redirected for several years (literally). Some kind of housekeeping?

Does "pages crawled per day" mean actual crawls, or requests? I'm looking at one site where the "crawls per day" is several times the total number of pages on the site-- and it definitely isn't the kind of site* where the same URL would be visited many times a day. The numbers only work if I count redirects (in this case, to a different site-- whose numbers haven't changed noticeably).


* Like, say, WebmasterWorld ;)