Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Want to Deny Google from Indexing, Crawling My Private Site

phranque - 11:38 pm on Aug 12, 2010 (gmt 0)


have you tried the robots.txt test function on the Crawler access page of Google Webmaster Tools?
https://www.google.com/webmasters/tools/crawl-access?hl=en&siteUrl=http://www.example.com/

Thread source:: http://www.webmasterworld.com/robots_txt/4185458.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com