Welcome to WebmasterWorld Guest from 18.104.22.168
Forum Moderators: goodroi
My robots.txt contains:
I put this txt file in my main directory
But when i type the name of my site i acn see in browsersomething like this:
where i must put the robots.txt in main directory or in /en directory?
10x in advance
The proper format is as follows
The following allows all robots to visit all files because the wildcard "*" specifies all robots.
This one keeps all robots out.
You want to place the robots.txt where yout html files are.
A robots.txt file will not help to get your site indexed in a search engine, this file is only used to disallow robots not to allow, and thats only if the spider or robot what may visit is a friendly robot, if its not it will just ignore the file and spider your site anyway
See this link below on how to build a successful site