| Spider Blocking
still a bit unclear |
flexsez
msg:1528637 | 6:06 pm on Oct 23, 2002 (gmt 0) |
I have read the robots.txt tutorial and I'm still a bit
unclear about where the robots.xtx file belongs or where
to find it. I do see that when a spider comes a crawlin' they seem
to find my robots.txt file just fine. The question is: where the heck is that file? The tutorial
says to put it in the root folder...does that mean the
robots.txt file should be included in the html of the index
page? P.S. I use dreamweaver 4 if that matters.
|
transistor
msg:1528638 | 7:05 pm on Oct 23, 2002 (gmt 0) |
The "root folder" almost always means the main directory of your site (that goes for every site you might host).
Let's say, you have a domain foo, and the "root folder" on your server is:
/dir1/dir2/dir3/foo/, now that's where robots.txt belongs to.
It sits right next to your main index.html (or whatever suffix you've got).
That's it. Oh, and beware of text editors, make your robots.txt file with Notepad or BBEdit (or maybe even use Pico to create the file on your server if it is some kind of Unix). If you have uploaded your robots.txt file and still in doubt, use the Robots.txt Validator at SearchEngineWorld [http].
|
amoore
msg:1528639 | 7:43 pm on Oct 23, 2002 (gmt 0) |
Another way to look at is that it's supposed to be at [example.com...] (where of course example.com is your domain). In fact, you can see other people's robots.txt files by looking at, say, [webmasterworld.com...] Hope it helps. -Andy
|
|
|
