Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Scooter


tschild - 5:54 pm on Jul 6, 2003 (gmt 0)


Subfolders should be covered by a Disallow directive.

Your robots.txt seems to be quite long at 2390 bytes, might a syntax error have crept in somewhere? There are several robots.txt syntax checkers out there.


Thread source:: http://www.webmasterworld.com/search_engine_spiders/2068.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com