Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Quick primer on identifying bot activity.
chandrika - 5:38 pm on Apr 2, 2008 (gmt 0)Thread source:: http://www.webmasterworld.com/search_engine_spiders/3613998.htm
Any thoughts on using an IPlist such as at [iplists.com...] to cloak a sitemap, as I have noticed that most scrapers are using my sitemaps in their scripts, If I disallow all except known spider IPs from viewing the sitemaps, might this help?
Brought to you by WebmasterWorld: http://www.webmasterworld.com