homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum


 4:11 am on Feb 11, 2003 (gmt 0)


I have a question about the robots.txt file.

First of all, does it HAVE to be there? What if its not there? To my best logic, everyone can browse thru the site then and the spiders can see EVERYTHING on my site(which i don't mind for now).

Also, is it the ROOT where the robots.txt go or WWW folder?

Thank You, all answers should help me. I am a learner.

Tx, Again.




 4:19 am on Feb 11, 2003 (gmt 0)

robots.txt goes in the document root of your domain, not the root of the server. It goes wherever the file www.yourdomain.com/index.html would go.

If there is no robots.txt, all spiders will feel welcome to index all of your files. The only bad effect of not having a robots.txt file is that your error log will have a lot of 404-Not Found errors cluttering it up as a result of spiders requesting robots.txt.

To avoid this, place a simple robots.txt file in your web root:

User-agent: *

With no text following "Disallow:" on the second line, this will welcome all robots to all files on your site.

You can then use this robots.txt checker [searchengineworld.com] on the WebmasterWorld sister site Search Engine World to help make sure that it is correct.



 4:24 am on Feb 11, 2003 (gmt 0)

JdMorgan I was trying to say the same thing but you do it so much better -- and much more quickly. Anyway...

Hi neh2008,

No, the robots.txt file does not have to be there. But you might save yourself some future questions (Why are there so many file not found errors for robots.tx), if you simply put a blank robots text file on the site.

Just open a file in Notepad or similar text editor and save it as robots.txt.

Where does it go? The exact location depends on your server but it in all cases it should be at the same level as your home page (index or default).

As you develop your site you might find instances where you would not want a robot indexing your pages. That's when you'll put the robots.txt to use.

When you need to, do a "site search" above to learn more than you really want to knwo about robts.txt.



 5:27 pm on Feb 11, 2003 (gmt 0)

Great answers, JDMorgan and JimBeetle....

I thank you both. It was helpful. When you do something for the first time in your life, you are skeptical. That's what has been happening to me.

The spider came and i had a 404 error in the logs too....

Thank you...

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved