Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

robot.txt blocked pages but still alive

10:19 pm on Sep 23, 2008 (gmt 0)

Preferred Member

10+ Year Member

joined:Apr 29, 2008
posts: 624
votes: 0


I did put a robot.txt file on my ftp server but what is bizzard is that if I type the address I have blocked in the txt file , I have a webpage that is currently live of my website that appears with this address ( in other words a page that is live on my site has now 2 address including one that I blocked in robot.txt) ?

My website is dynamic ? maybe that is an issue ?

Can anyone can explain where this comes from, how to fix it and if it is an issue for ranking on google.


1:34 pm on Sept 25, 2008 (gmt 0)

Administrator from US 

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:June 21, 2004
votes: 290

You should make sure to place the robots.txt file in the root of your web folder. FTP servers have many folders and if you place the robots.txt file in the wrong one it will not work.

Robots.txt helps to control how search engine robots access your website. It will not restrict your pages from appearing in internet browsers.

When you say your pages are appearing at two addresses are you referring to http://www.example.com and http://example.com both showing the same content? If that is the situation you should use htaccess to apply a 301 redirect so the content is only available from one source. A 301 redirect will redirect users from the other source and automatically send them to your one source.

1:27 pm on Oct 21, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
votes: 0

Robots.txt does not stop you accessing a URL. It cannot block access if something requests that URL.

It is a file that compliant bots download from your site, and then use to make a note of URLs that they should not request.

Robots.txt doesn't actually "block" anything at all. It is merely a list of questions that should not be asked.