Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

robot.txt blocked pages but still alive



10:19 pm on Sep 23, 2008 (gmt 0)

5+ Year Member


I did put a robot.txt file on my ftp server but what is bizzard is that if I type the address I have blocked in the txt file , I have a webpage that is currently live of my website that appears with this address ( in other words a page that is live on my site has now 2 address including one that I blocked in robot.txt) ?

My website is dynamic ? maybe that is an issue ?

Can anyone can explain where this comes from, how to fix it and if it is an issue for ranking on google.



1:34 pm on Sep 25, 2008 (gmt 0)

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

You should make sure to place the robots.txt file in the root of your web folder. FTP servers have many folders and if you place the robots.txt file in the wrong one it will not work.

Robots.txt helps to control how search engine robots access your website. It will not restrict your pages from appearing in internet browsers.

When you say your pages are appearing at two addresses are you referring to http://www.example.com and http://example.com both showing the same content? If that is the situation you should use htaccess to apply a 301 redirect so the content is only available from one source. A 301 redirect will redirect users from the other source and automatically send them to your one source.


1:27 pm on Oct 21, 2008 (gmt 0)

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

Robots.txt does not stop you accessing a URL. It cannot block access if something requests that URL.

It is a file that compliant bots download from your site, and then use to make a note of URLs that they should not request.

Robots.txt doesn't actually "block" anything at all. It is merely a list of questions that should not be asked.


Featured Threads

Hot Threads This Week

Hot Threads This Month