homepage Welcome to WebmasterWorld Guest from 54.243.13.30
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld

Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Robots.txt
I better get this right...
JVB_Mktg




msg:1525555
 5:53 pm on Feb 9, 2003 (gmt 0)

I want to add the robots.txt file to my site, but this is something that I really want to get right.

What would the basic file look like without excluding anything.

TIA,
Javi

 

heini




msg:1525556
 6:30 pm on Feb 9, 2003 (gmt 0)

>basic
User-agent: * 
Disallow:

A great and very easy to understand tutorial is here:
[searchengineworld.com...]

And welcome to the board!

JVB_Mktg




msg:1525557
 7:57 pm on Feb 9, 2003 (gmt 0)

Thanks for the link Heini. :)

Let's see if I undertand,

User-agent: *
Disallow: /forum/

Would just keep the spiders off my forum right? Everything else would be found.

Javi :)

heini




msg:1525558
 8:16 pm on Feb 9, 2003 (gmt 0)

Yup, that looks correct to me.
Everything else is allowed to be retrieved by any visitor, be it bot or browser.

neh2008




msg:1525559
 4:16 am on Feb 11, 2003 (gmt 0)

my question is,

where does the file go?

in the root directory? of the WWW root?

my dir structure is....

/
www cgi-bin images
[all html content]

should i put robots.txt here with the HTML content? or one level up?

thanks

Jack_Straw




msg:1525560
 4:19 am on Feb 11, 2003 (gmt 0)

It goes in the same directory as your index or default file. Most people would call that the www root directory, I think.

Test with: [mydomain.com...]

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved