Welcome to WebmasterWorld Guest from 184.108.40.206
Forum Moderators: open
Here's what I've been seeing: a small (test?) crawl in September (9/7-12), a full crawl in late September (9/21-10/3), another crawl (with FAST-WebCrawler/3.3) in mid-October (10/19-26)), and what seems like a full crawl starting yesterday (11/13). So, basically one crawl every 4 weeks.
What is everyone else seeing? Do I have to do something special to be refreshed more often? Should I submit my URLs every 9-12 days?
all my sites prior to october get spidered on a regular basis, and a great number of these sites have links to my post october sites, but no joy.
we don't use a robot.txt file to exclude any spiders.