Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
mack - 10:57 am on Jul 6, 2003 (gmt 0)Thread source:: http://www.webmasterworld.com/search_engine_spiders/2068.htm
Scooter seams to respect robots.txt in my sites. Unusual for it to disrespect. Not being funny but are you sure you used the correct syntax in your robots.txt file?
Brought to you by WebmasterWorld: http://www.webmasterworld.com