Page is a not externally linkable
- Google
-- Google SEO News and Discussion
---- Pages are indexed even after blocking in robots.txt


tedster - 7:09 am on Sep 6, 2012 (gmt 0)


I use robots.txt to disallow certain URL patterns simply by showing the beginning of the pattern.

So, I can easily exclude various parameters that might lead googlebot into an major duplicate content area, such as:

Disallow /category?sort=

Google has listened to that every time I've used it, and I've used it a LOT.


Thread source:: http://www.webmasterworld.com/google/4490125.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com