Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Sitemaps to Google, Yahoo and Bing, am I missing another one?
phranque - 2:07 am on Jul 20, 2009 (gmt 0)Thread source:: http://www.webmasterworld.com/robots_txt/3945367.htm
any "well-behaved" robot will request and respect robots.txt before crawling your site.
some bots will check robots.txt before each page request and others will do it once per "session".
Brought to you by WebmasterWorld: http://www.webmasterworld.com