Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- abot/0.1


jim_w - 8:02 pm on May 9, 2005 (gmt 0)


Has anyone seen this?

24.177.134.6 - - [07/May/2005:03:08:02 -0400] "GET /robots.txt HTTP/1.0" 403 4429 "-" "aipbot/1.0 (aipbot; [aipbot.com;...] aipbot@aipbot.com)"

202.96.140.119 - - [07/May/2005:04:14:44 -0400] "GET / HTTP/1.0" 403 4429 "-" "AIBOT/2.1 By +(www.21seek.com , A Real artificial intelligence search engine , China)"

I'm confused. Ban both blocks? Are both scrapers, or what ever you want to call them, et. al.


Thread source:: http://www.webmasterworld.com/search_engine_spiders/2799.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com