Why are non search engines robots hitting our site?
Acternaweb
8:13 pm on Feb 14, 2001 (gmt 0)
This is probably a dumb question, but in viewing the Robot/Spider report on Net Tracker there are non search engine spiders there. I go to their site, by our content is not found. We do not have a a relationship with them.
Thanks,
Paul
rogerd
4:27 pm on Feb 15, 2001 (gmt 0)
The web is filled with robots that aren't from search engines. Some are e-mail harvesters, others content archivers, still others experimental bots or bots gathering content for some purpose known only to the operator.
You can try to exclude specific bots with your robots.txt file if you want to try, although nefarious bots like e-mail harvesters will probably pay no heed.