Welcome to WebmasterWorld Guest from 22.214.171.124
Forum Moderators: goodroi
joined:Feb 3, 2004
1) is it wrong to have a robots.txt in two dif locations?
Robots.txt is expected to live off the root of the website, you can try putting it in other places but it wont get used.
2) is this proper syntax
Depends what you hope to achieve - the snippet you supplied is the disallow portion of robots.txt, but you also need a user-agent filter as well - a very simple example that applies to all robots would be;
3) I know most Spiders have a default of index follow, but Is it ok if I place that meta in every page or should I only place it once in the home page?
You can place robots metatags on every page if you want to, although since most commercial spiders default to "index follow" you don't need it in order to get crawled. Like all meta tag data they apply only to the current page so you can't just place it only on the front page and expect it to propagate through the site.
I just copy the question I post this morning as it is related to this topic :
1/ I am working on a beta version of my website and will upload it on a subdomain (beta.DNS.com) just to try it.
I do not want robots to crawl the site. If I put a robots.txt file in the root of the directory (with User-agent: * and Disallow: /), will robots ignore the site despite I have on each file a meta tag CONTENT="all".
I do not want robots to crawl the site.
Then you might want to put the site behind password protection. Both methods of robots directives are not foolproof. The robots.txt file at the root level is more reliable than the META robots tag.
Keep in mind that not all robots follow the protocol. Many will skip the robots.txt file and go right through your site.