Welcome to WebmasterWorld Guest from 184.108.40.206
Forum Moderators: goodroi
I e-mailed my web host asking where mine was located on the server and they told me they never use one. Should I be trying to make one for the SE's and where should I upload it? Thanks,
Most SE's will/should obey a robots.txt file while there is a lesser amount will read the META tag.
Am open for correction on this, but I think that is what i have came to understand.
If you have an META tag instruction for robots, do you need a robots.txt?
At the very least, you should put an empty robots.txt to cut down on 404s on your site.
Using META tags allows you to control how robots index or follow links on your pages.
Using robots.txt restricts access to files (text and binary), as well as directories.
If nobody knows, nobody will make a law about it.