mojeek - 7:47 am on May 1, 2012 (gmt 0)

Hi jmcc, I think you're spot on. Blindly crawling everything in sight is definitely not the way to go for smaller engines, it's then just a matter of deciding what to crawl or not, either algorithmically or manually.