Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Spider Blocking

still a bit unclear

6:06 pm on Oct 23, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:Aug 30, 2002
votes: 0

I have read the robots.txt tutorial and I'm still a bit
unclear about where the robots.xtx file belongs or where
to find it.

I do see that when a spider comes a crawlin' they seem
to find my robots.txt file just fine.

The question is: where the heck is that file? The tutorial
says to put it in the root folder...does that mean the
robots.txt file should be included in the html of the index

P.S. I use dreamweaver 4 if that matters.

7:05 pm on Oct 23, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:May 2, 2002
votes: 0

The "root folder" almost always means the main directory of your site (that goes for every site you might host).
Let's say, you have a domain foo, and the "root folder" on your server is:
/dir1/dir2/dir3/foo/, now that's where robots.txt belongs to.
It sits right next to your main index.html (or whatever suffix you've got).
That's it.

Oh, and beware of text editors, make your robots.txt file with Notepad or BBEdit (or maybe even use Pico to create the file on your server if it is some kind of Unix).

If you have uploaded your robots.txt file and still in doubt, use the Robots.txt Validator at SearchEngineWorld [http].

7:43 pm on Oct 23, 2002 (gmt 0)

Preferred Member

10+ Year Member

joined:Nov 30, 2001
votes: 0

Another way to look at is that it's supposed to be at [example.com...] (where of course example.com is your domain). In fact, you can see other people's robots.txt files by looking at, say, [webmasterworld.com...]

Hope it helps.