Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Google Web Preview


londrum - 9:08 pm on Nov 10, 2010 (gmt 0)


its a bit worrying for anyone with a bot trap.

if you're currently blocking spiders from crawling stuff like images and javascript in robots.txt then they are obviously ignoring it. because i am doing that to... yet the picture shows all the images and javascript-created text intact.

which means the spider could also crawl the bot trap and get sprung.


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4223018.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com