Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- ia archiver


wilderness - 3:16 pm on Jul 2, 2012 (gmt 0)


Obviously this varies for all of us depending on niche, so it comes down to a business decisions pro/con. Quite similar to Websense - sites which rely heavily on corporate traffic may not want to block it while others may find it beneficial.


Given today's market trends and the variety of devices (PC's, cell-phones, Handhelds), I'm seeing legitimate users simply switching devices.

Course that's not economical for the mass harvester, nor are the bandwidth restrictions of most mobile devices.


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4471054.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com