Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- How do I stop g bot from crawling directories?


g1smd - 5:38 pm on Jan 8, 2012 (gmt 0)


automatically redirect to the same url with the /gl/ removed, is the pesky duplicate content

simply delete some folders/pages and redirect them to the old English only version without worrying about such nonsense like dup content

Redirects are not duplicate content.

Duplicate content is same content served with 200 OK status at multiple URLs.

The danger of "redirecting everything" is "infinite URL space".


Thread source:: http://www.webmasterworld.com/robots_txt/4368085.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com