Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- robot.txt blocked pages but still alive


goodroi - 1:34 pm on Sep 25, 2008 (gmt 0)


You should make sure to place the robots.txt file in the root of your web folder. FTP servers have many folders and if you place the robots.txt file in the wrong one it will not work.

Robots.txt helps to control how search engine robots access your website. It will not restrict your pages from appearing in internet browsers.

When you say your pages are appearing at two addresses are you referring to http://www.example.com and http://example.com both showing the same content? If that is the situation you should use htaccess to apply a 301 redirect so the content is only available from one source. A 301 redirect will redirect users from the other source and automatically send them to your one source.


Thread source:: http://www.webmasterworld.com/robots_txt/3750825.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com