Page is a not externally linkable
- Google
-- Google SEO News and Discussion
---- Pages are indexed even after blocking in robots.txt


shaunm - 7:33 am on Sep 6, 2012 (gmt 0)


@tedster
Could you please tell me what do you mean by 'Disallow' here?

So, I can easily exclude various parameters that might lead googlebot into an major duplicate content area, such as:

Disallow /category?sort=


Are you sure that this patterning works? This is all new to me. Because I only heard that you should put the exact path(after the FIST SLASH till the end) to prevent a page/folder/file from getting crawled. Thanks


Thread source:: http://www.webmasterworld.com/google/4490125.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com