Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- amazonaws.com plays host to wide variety of bad bots
thetrasher - 12:32 pm on Feb 16, 2009 (gmt 0)Thread source:: http://www.webmasterworld.com/search_engine_spiders/3828718.htm
Sorry. What I tried to write:
reads robots.txt, but doesn't care about the contents of robots.txt.
|Nokia6680/1.0 (4.04.07) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 Configuration/CLDC-1.1 (botmobi find.mobi/bot.html email@example.com) |
Really bad bots request for robots.txt in order to get into the dark web and to confuse webmasters ("robots.txt? YES").
|To my recollection, most that asked for robots.txt honored it. |
Requesting my robots.txt leads to a site-wide ban.
|Technically, there's nothing in robots.txt that prevents any bot from doing whatever the heck its runners program it to do. |
Brought to you by WebmasterWorld: http://www.webmasterworld.com