homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

new to robots
is this right?

 6:14 pm on Apr 3, 2004 (gmt 0)

Hi, I have been reading a lot and added a robots.txt to my root directory yesterday snipped and it validates fine. Now, my question is, do I have it in the right spot?

and on every page of my site I have the meta tag

<META NAME="robots" CONTENT="index,follow">

is this right?

In my root directory, there is one index.html file which is our site, but at the beginning of building it, this was a splash page (not any more) so I had the first page as index2.htm. I did not want to submit the splash page to search engines and such so I submitted the index2.htm page (which is indexed) and so now I have that page in my root directory also, along with one more page and some js files.

My main site is in a directory called page. Do I have this set right for the search engine or robots to crawl this site? Or do I need to change something?


[edited by: DaveAtIFG at 6:11 pm (utc) on April 5, 2004]
[edit reason] Url removed [/edit]



 6:26 pm on Apr 3, 2004 (gmt 0)

Looks ok to me, also I've never used the robots follow tag on my webpages as the crawlers follow without them.

PS. Thanks for the link to the midifiles, you may want to put a blank index.html in those folders.


 6:30 pm on Apr 3, 2004 (gmt 0)

Hi, You're welcome for the midi's What do you mean about a blank index.html? And in what folders? Please explain in more detail if it is not too much trouble.


 6:33 pm on Apr 3, 2004 (gmt 0)

If you wanted to stop people looking at the content of the midi folder save a blank file as index.html and ftp it up to the midi folder, then when people find yoursitedotcom/midi/ they get presented with a blank page instead of a listing of the foleders contents.


 6:43 pm on Apr 3, 2004 (gmt 0)

Ok, I think I get it,

and should I do this in the other directory's also that I would not want people to see?

What about a doc tag

<meta name="resource-type" content="document">

someone on another board told me I need a doc tag. Is that a doc tag above and do I need it?


 5:38 pm on Apr 5, 2004 (gmt 0)

Document headers are not essential, I use them sometimes but in my opinion they make no difference, a typical header would be;


If you want to exclude crawlers modify your robots.txt file, if you don't have one create one in notepad, save it as robots.txt then ftp it up to your site, example below;

User-agent: *
Disallow: /tmp
Disallow: /logs

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved