Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- How do I stop g bot from crawling directories?


shazam - 12:00 pm on Sep 29, 2011 (gmt 0)


Thanks for the suggestions.

I would rather just block ALL traffic, not just the bots. It may appear to be cloaking if they do a manual inspection.

The whole thing is just silly. I guy should be able to simply delete some folders/pages and redirect them to the old English only version without worrying about such nonsense like dup content or cloaking. These days almost whatever you do will get penalized in one form or another. I miss the days when we just had to make good websites with quality content and the search engines followed along.

It's also starting to look like robots.txt files are a complete waste of time.


Thread source:: http://www.webmasterworld.com/robots_txt/4368085.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com