Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Charlotte the spider
keyplyr - 11:45 pm on May 22, 2006 (gmt 0)Thread source:: http://www.webmasterworld.com/search_engine_spiders/3252.htm
|What are these guys looking for by deliberately requesting URLs that don't exist? |
Seems to be a new scraping technique, at least in part. I am seeing this from several different known scraper "directories."
Brought to you by WebmasterWorld: http://www.webmasterworld.com