Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- amazonaws.com plays host to wide variety of bad bots

thetrasher - 9:54 pm on Feb 15, 2009 (gmt 0)


Nokia6680/1.0 (4.04.07) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 Configuration/CLDC-1.1 (botmobi find.mobi/bot.html find@mtld.mobi)

robots.txt? YES

You may write whatever you want into your robots.txt.
User-agent: *
Disallow: /

doesn't prevent requesting the root page.

Thread source:: http://www.webmasterworld.com/search_engine_spiders/3828718.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com