Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
blend27 - 12:50 am on Apr 23, 2013 (gmt 0)Thread source:: http://www.webmasterworld.com/search_engine_spiders/4566791.htm
The point is that more and more AJAX based scarper scripts are out there being released into the wild. At this point HEADERS are very effective against that. Node.js is a serverside JS technology BTW.
Brought to you by WebmasterWorld: http://www.webmasterworld.com