Robots - Allowing them

rather than disallow

[SanDiegoPaul]

First: is it true that you have to have the robots.txt file in the server ROOT? Or just in the root of your website? On one of my servers, I'm required to put my files and folders WITHIN an /html/ folder under my domain. Can I put a robots.txt file in there?

Next what if I want to do it within html code, rather than a robots.txt file? is it just a simple code such as

<meta name="robots" content="follow">

would that do it? If so then how would you do exclusions within html?

thanks for any feedback

Paul in San Diego

[jatar_k]

the robots.txt goes in the web root, not the server root

it needs to be at this location

www.yoursite.com/robots.txt

if you want pages to be followed and have no exclusions then just have a blank robots file. You shouldn't to exclude on page as the robots.txt should work just fine. For those spiders that don't follow it they won't follow the html style either.

a tag that excludes would look something like this

<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

though mainly the robots tag is for people who can;t control/use a robots.txt