homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Spider Blocking
still a bit unclear

 6:06 pm on Oct 23, 2002 (gmt 0)

I have read the robots.txt tutorial and I'm still a bit
unclear about where the robots.xtx file belongs or where
to find it.

I do see that when a spider comes a crawlin' they seem
to find my robots.txt file just fine.

The question is: where the heck is that file? The tutorial
says to put it in the root folder...does that mean the
robots.txt file should be included in the html of the index

P.S. I use dreamweaver 4 if that matters.



 7:05 pm on Oct 23, 2002 (gmt 0)

The "root folder" almost always means the main directory of your site (that goes for every site you might host).
Let's say, you have a domain foo, and the "root folder" on your server is:
/dir1/dir2/dir3/foo/, now that's where robots.txt belongs to.
It sits right next to your main index.html (or whatever suffix you've got).
That's it.

Oh, and beware of text editors, make your robots.txt file with Notepad or BBEdit (or maybe even use Pico to create the file on your server if it is some kind of Unix).

If you have uploaded your robots.txt file and still in doubt, use the Robots.txt Validator at SearchEngineWorld [http].


 7:43 pm on Oct 23, 2002 (gmt 0)

Another way to look at is that it's supposed to be at [example.com...] (where of course example.com is your domain). In fact, you can see other people's robots.txt files by looking at, say, [webmasterworld.com...]

Hope it helps.


Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved