dataguy - 8:54 pm on Oct 17, 2006 (gmt 0)

Great thread, Webwork.

I'm the owner/developer of a directory which I've completely custom-coded. We have geo-targeted listings broken out over about 80,000 cities world-wide. We receive about 15,000 submissions per day, and by using a combination of pre-crawling, algo-filtering, caching and human review we typically whittle this down to about 2,000 new listings per day. I've also created a thumbnail crawler which captures screen shots of the web sites for the listings.