Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Yahoo! Slurp China


kevinpate - 2:50 am on Dec 1, 2005 (gmt 0)


I have only one heartburn to the continuous serving up of 403's to Slurp China, and it's not at all unique to that one bot.

What possibly bennie is there for any bot to come back and hit a 403 multi times a day/week/month/etc.? It's like every bot has a line of code that says never give up, never surrender.


Thread source:: http://www.webmasterworld.com/search_engine_spiders/3086.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com