Welcome to WebmasterWorld Guest from 18.104.22.168
Forum Moderators: goodroi
I did put a robot.txt file on my ftp server but what is bizzard is that if I type the address I have blocked in the txt file , I have a webpage that is currently live of my website that appears with this address ( in other words a page that is live on my site has now 2 address including one that I blocked in robot.txt) ?
My website is dynamic ? maybe that is an issue ?
Can anyone can explain where this comes from, how to fix it and if it is an issue for ranking on google.
Robots.txt helps to control how search engine robots access your website. It will not restrict your pages from appearing in internet browsers.
When you say your pages are appearing at two addresses are you referring to http://www.example.com and http://example.com both showing the same content? If that is the situation you should use htaccess to apply a 301 redirect so the content is only available from one source. A 301 redirect will redirect users from the other source and automatically send them to your one source.
It is a file that compliant bots download from your site, and then use to make a note of URLs that they should not request.
Robots.txt doesn't actually "block" anything at all. It is merely a list of questions that should not be asked.