Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- At Home with the Robots
incrediBILL - 1:24 am on Feb 12, 2012 (gmt 0)Thread source:: http://www.webmasterworld.com/search_engine_spiders/4416720.htm
|My htaccess was getting fatter and the Ignore list was getting longer. |
That's because you don't whitelist.
Blacklisting is bulky, time consuming and inefficient.
Not to mention it puts a bigger load on the server.
Brought to you by WebmasterWorld: http://www.webmasterworld.com