Today, my main site experienced a deep crawl by looksmart. The robot used an IP address which resolved to *.looksmart.com and spent the better part of the day crawling the site.
As a matter of fact, my site(s) are already listed with all major keywords, but the lesser keywords and the "poison pills" from the lower pages are not to be found in their index.
So what just happened today?
pmkpmk
7:54 am on Apr 27, 2005 (gmt 0)
It's still crawling. The spider's name is "LookSmart", and it comes from sv-crawl.looksmart.com and sv-crawlfw3.looksmart.com
seth_wilde
3:23 pm on Apr 27, 2005 (gmt 0)
Sounds like it was problably Wisenut (which Looksmart owns).. It's a legit crawler engine which was actually showing some promise back in 2001 before looksmart bought them. I believe Looksmart also spiders sites in their directory to find dead links.
