Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

new to robots

is this right?



6:14 pm on Apr 3, 2004 (gmt 0)

10+ Year Member

Hi, I have been reading a lot and added a robots.txt to my root directory yesterday snipped and it validates fine. Now, my question is, do I have it in the right spot?

and on every page of my site I have the meta tag

<META NAME="robots" CONTENT="index,follow">

is this right?

In my root directory, there is one index.html file which is our site, but at the beginning of building it, this was a splash page (not any more) so I had the first page as index2.htm. I did not want to submit the splash page to search engines and such so I submitted the index2.htm page (which is indexed) and so now I have that page in my root directory also, along with one more page and some js files.

My main site is in a directory called page. Do I have this set right for the search engine or robots to crawl this site? Or do I need to change something?


[edited by: DaveAtIFG at 6:11 pm (utc) on April 5, 2004]
[edit reason] Url removed [/edit]


6:26 pm on Apr 3, 2004 (gmt 0)

10+ Year Member

Looks ok to me, also I've never used the robots follow tag on my webpages as the crawlers follow without them.

PS. Thanks for the link to the midifiles, you may want to put a blank index.html in those folders.


6:30 pm on Apr 3, 2004 (gmt 0)

10+ Year Member

Hi, You're welcome for the midi's What do you mean about a blank index.html? And in what folders? Please explain in more detail if it is not too much trouble.


6:33 pm on Apr 3, 2004 (gmt 0)

10+ Year Member

If you wanted to stop people looking at the content of the midi folder save a blank file as index.html and ftp it up to the midi folder, then when people find yoursitedotcom/midi/ they get presented with a blank page instead of a listing of the foleders contents.


6:43 pm on Apr 3, 2004 (gmt 0)

10+ Year Member

Ok, I think I get it,

and should I do this in the other directory's also that I would not want people to see?

What about a doc tag

<meta name="resource-type" content="document">

someone on another board told me I need a doc tag. Is that a doc tag above and do I need it?


5:38 pm on Apr 5, 2004 (gmt 0)

10+ Year Member

Document headers are not essential, I use them sometimes but in my opinion they make no difference, a typical header would be;


If you want to exclude crawlers modify your robots.txt file, if you don't have one create one in notepad, save it as robots.txt then ftp it up to your site, example below;

User-agent: *
Disallow: /tmp
Disallow: /logs


Featured Threads

Hot Threads This Week

Hot Threads This Month