Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- EdisterBot


lucy24 - 12:19 pm on Feb 25, 2012 (gmt 0)


they don't know who you are initially so there is no disallow to honor, ever.

? Is your site so wide-open that you never have to use a generic "Hey you" ?

User-Agent: *
Disallow: /keepout
Disallow: /getlost
Disallow: /OKhere/dontgohere

Doesn't take much for a robot to honor or dishonor something in that form. I once had an under-construction site with a phony directory set up purely to trap robots ahead of time.


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4421083.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com