homepage Welcome to WebmasterWorld Guest from 54.166.122.65
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / WebmasterWorld / Website Analytics - Tracking and Logging
Forum Library, Charter, Moderators: Receptional & mademetop

Website Analytics - Tracking and Logging Forum

    
Spidering Slow Down
Daily spidering now only weekly
webseosolution



 
Msg#: 4597601 posted 7:52 am on Jul 30, 2013 (gmt 0)

Dear All,

My website home page was being crawled on daily basis earlier.. but now it takes at least 7 days.. Can any one let me know that what reason should behind that?




Thanks,
WSS.

 

lucy24

WebmasterWorld Senior Member lucy24 us a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month



 
Msg#: 4597601 posted 9:43 pm on Jul 30, 2013 (gmt 0)

How often does its content change? How much search traffic does it get?

mademetop

WebmasterWorld Administrator 10+ Year Member



 
Msg#: 4597601 posted 8:49 am on Jul 31, 2013 (gmt 0)

The reasons can be multiple and most are perfectly logical. If you are not updating your home page on a daily basis - why should a spider waste bandwidth checking on a daily basis?

Most SE spiders are looking for new content on pages that are frequently changed or seeking links to new pages that have been created. So, as is usual for static home pages, crawl frequency can be reduced as there is likely to be nothing new (or of critical importance) to add to the SE database.

webseosolution



 
Msg#: 4597601 posted 1:19 pm on Aug 2, 2013 (gmt 0)

Content changes on my website on daily basis, and its getting arround 1000 visitors every day.

lucy24

WebmasterWorld Senior Member lucy24 us a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month



 
Msg#: 4597601 posted 8:10 pm on Aug 2, 2013 (gmt 0)

Content changes on my website

Not on your website as a whole. On the specific, individual pages that are being crawled. Are changes visible on the front page?

tangor

WebmasterWorld Senior Member tangor us a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



 
Msg#: 4597601 posted 10:05 pm on Aug 2, 2013 (gmt 0)

There's a side benefit with the "static" or "evergreen" pages in reduction of bot activity, but the SITE itself should show new content/pages on a routine basis to keep the bots coming back. The home page should show change...

How long has the site been up, and what time period has elapsed before the bots slowed down?

webseosolution



 
Msg#: 4597601 posted 7:09 am on Aug 5, 2013 (gmt 0)

To Lucy - yes all the fresh content are visible on home page.

To Tangor - The changes are being shown on home page on daily basis...

The site is up since 2006, Elapsed from almost 1.5 month

ronalds8



 
Msg#: 4597601 posted 7:48 am on Aug 5, 2013 (gmt 0)

If you get hit by panda or penguin, spider slows down also.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / WebmasterWorld / Website Analytics - Tracking and Logging
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved