homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

robot.txt blocked pages but still alive

 10:19 pm on Sep 23, 2008 (gmt 0)


I did put a robot.txt file on my ftp server but what is bizzard is that if I type the address I have blocked in the txt file , I have a webpage that is currently live of my website that appears with this address ( in other words a page that is live on my site has now 2 address including one that I blocked in robot.txt) ?

My website is dynamic ? maybe that is an issue ?

Can anyone can explain where this comes from, how to fix it and if it is an issue for ranking on google.




 1:34 pm on Sep 25, 2008 (gmt 0)

You should make sure to place the robots.txt file in the root of your web folder. FTP servers have many folders and if you place the robots.txt file in the wrong one it will not work.

Robots.txt helps to control how search engine robots access your website. It will not restrict your pages from appearing in internet browsers.

When you say your pages are appearing at two addresses are you referring to http://www.example.com and http://example.com both showing the same content? If that is the situation you should use htaccess to apply a 301 redirect so the content is only available from one source. A 301 redirect will redirect users from the other source and automatically send them to your one source.


 1:27 pm on Oct 21, 2008 (gmt 0)

Robots.txt does not stop you accessing a URL. It cannot block access if something requests that URL.

It is a file that compliant bots download from your site, and then use to make a note of URLs that they should not request.

Robots.txt doesn't actually "block" anything at all. It is merely a list of questions that should not be asked.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved