grandma_genie - 4:12 pm on Aug 16, 2010 (gmt 0)
I found this new (to me anyway) crawler in my server logs. They just checked out robots.txt. User agent was voyager/2.0. IP is from www.kosmix.com/crawler.html. IP is 18.104.22.168. Does anyone have any comments about this crawler? Should they be allowed. I have a terrible time with bots coming to my image rich site and not only taking the images, but also the content. I find my content in the strangest places.