Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- MSN's many cloaked bots. Again.

Pfui - 12:00 am on Oct 27, 2010 (gmt 0)


On my Apache server, all /logs directories and /access_log files are 'above' web spaces. Thus there's no way anyone, or anything, should be able to access them from the web, neither is there any reason to include them in robots.txt. That MSN is even looking for /logs or /access_logs/ is outrageous.

Thread source:: http://www.webmasterworld.com/search_engine_spiders/4182830.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com