Effective Page Refresh Policies For Web Crawlers

[vitaplease]

Effective Page Refresh Policies For Web Crawlers [dbpubs.stanford.edu]

JUNGHOO CHO - University of California, Los Angeles
and
HECTOR GARCIA-MOLINA - Stanford University

A long read for those interested in how Search engine crawlers might operate to collect fresh pages most efficiently..

Our optimal policies consider how often a page changes and how important the pages are, and make an appropriate refresh decision.