Page is a not externally linkable
- Google
-- Google SEO News and Discussion
---- robots.txt - Google's JohnMu Tweets a tip


g1smd - 5:25 pm on May 30, 2010 (gmt 0)


No. The URLs disallowed in robots.txt are not crawled.

The URLs are discovered when they are published in the links found on other pages.

Those URLs are listed as URL-only entries in the SERPs because the content is never crawled or indexed.


Thread source:: http://www.webmasterworld.com/google/4143083.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com