Page is a not externally linkable
Glitzer - 4:41 pm on Jan 4, 2007 (gmt 0)
If they already have pages indexed or cached that have recently been disallowed in a revised robots.txt, then they won't remove those pages the SE's already indexed. Quite a dilemma for webmasters.
Well-behaved robots, including those from major search engines, will not fetch a page if it is Disallowed in a properly-formatted robots.txt file.