Welcome to WebmasterWorld Guest from 54.160.254.203

Forum Moderators: DixonJones & mademetop

Message Too Old, No Replies

Spidering Slow Down

Daily spidering now only weekly

     
7:52 am on Jul 30, 2013 (gmt 0)

New User

joined:Aug 20, 2012
posts:26
votes: 0


Dear All,

My website home page was being crawled on daily basis earlier.. but now it takes at least 7 days.. Can any one let me know that what reason should behind that?




Thanks,
WSS.
9:43 pm on July 30, 2013 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month

joined:Apr 9, 2011
posts:12711
votes: 244


How often does its content change? How much search traffic does it get?
8:49 am on July 31, 2013 (gmt 0)

Moderator This Forum

WebmasterWorld Administrator 10+ Year Member

joined:July 21, 2000
posts:1476
votes: 0


The reasons can be multiple and most are perfectly logical. If you are not updating your home page on a daily basis - why should a spider waste bandwidth checking on a daily basis?

Most SE spiders are looking for new content on pages that are frequently changed or seeking links to new pages that have been created. So, as is usual for static home pages, crawl frequency can be reduced as there is likely to be nothing new (or of critical importance) to add to the SE database.
1:19 pm on Aug 2, 2013 (gmt 0)

New User

joined:Aug 20, 2012
posts:26
votes: 0


Content changes on my website on daily basis, and its getting arround 1000 visitors every day.
8:10 pm on Aug 2, 2013 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month

joined:Apr 9, 2011
posts:12711
votes: 244


Content changes on my website

Not on your website as a whole. On the specific, individual pages that are being crawled. Are changes visible on the front page?
10:05 pm on Aug 2, 2013 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 29, 2005
posts:6151
votes: 282


There's a side benefit with the "static" or "evergreen" pages in reduction of bot activity, but the SITE itself should show new content/pages on a routine basis to keep the bots coming back. The home page should show change...

How long has the site been up, and what time period has elapsed before the bots slowed down?
7:09 am on Aug 5, 2013 (gmt 0)

New User

joined:Aug 20, 2012
posts:26
votes: 0


To Lucy - yes all the fresh content are visible on home page.

To Tangor - The changes are being shown on home page on daily basis...

The site is up since 2006, Elapsed from almost 1.5 month
7:48 am on Aug 5, 2013 (gmt 0)

New User

joined:Sept 14, 2011
posts:17
votes: 0


If you get hit by panda or penguin, spider slows down also.