Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- possible botnet


tangor - 2:35 am on Jul 12, 2012 (gmt 0)


Been there, done that, had the panic, too... but then again, robots.txt is for everyone (or should be) and that's the way I roll on it (robots.txt). Get ugly with anything else on my site... well that's a different story! :)


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4474787.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com