Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- amazonaws.com plays host to wide variety of bad bots

jdMorgan - 1:16 am on Jan 21, 2009 (gmt 0)


Pfui,

Very nice list!

But for clarity, what is the meaning of the "Robots.txt? Yes/No" entries above?

The user-agent fetches robots.txt
-or-
The user-agent fetches and obeys robots.txt

Thanks,
Jim

Thread source:: http://www.webmasterworld.com/search_engine_spiders/3828718.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com