Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Hiding robots.txt


wackybrit - 10:51 pm on Oct 30, 2003 (gmt 0)


A "good" leeching bot (in the eye of the leecher) will disguise its UA and never obeye a robots.txt.

And the best have vast networks so they can appear to be coming from 100s of different IPs, with different UserAgents every time.. much like regular visitors.

I'm not sure if there's a viable way to block this sort of activity, as it's very hard to track.


Thread source:: http://www.webmasterworld.com/robots_txt/74.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com