Forum Moderators: open

Message Too Old, No Replies

Why are non search engines robots hitting our site?

         

Acternaweb

8:13 pm on Feb 14, 2001 (gmt 0)

10+ Year Member



This is probably a dumb question, but in viewing
the Robot/Spider report on Net Tracker there are non search engine spiders there. I go to their site, by our content is not found. We do not have a a relationship with them.

Thanks,

Paul

rogerd

4:27 pm on Feb 15, 2001 (gmt 0)

WebmasterWorld Administrator 10+ Year Member



The web is filled with robots that aren't from search engines. Some are e-mail harvesters, others content archivers, still others experimental bots or bots gathering content for some purpose known only to the operator.

You can try to exclude specific bots with your robots.txt file if you want to try, although nefarious bots like e-mail harvesters will probably pay no heed.