Hi, I have been reading a lot and added a robots.txt to my root directory yesterday snipped and it validates fine. Now, my question is, do I have it in the right spot?
and on every page of my site I have the meta tag
<META NAME="robots" CONTENT="index,follow">
is this right?
In my root directory, there is one index.html file which is our site, but at the beginning of building it, this was a splash page (not any more) so I had the first page as index2.htm. I did not want to submit the splash page to search engines and such so I submitted the index2.htm page (which is indexed) and so now I have that page in my root directory also, along with one more page and some js files.
My main site is in a directory called page. Do I have this set right for the search engine or robots to crawl this site? Or do I need to change something?
[edited by: DaveAtIFG at 6:11 pm (utc) on April 5, 2004] [edit reason] Url removed [/edit]
If you wanted to stop people looking at the content of the midi folder save a blank file as index.html and ftp it up to the midi folder, then when people find yoursitedotcom/midi/ they get presented with a blank page instead of a listing of the foleders contents.