Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Spider Blocking


flexsez - 6:06 pm on Oct 23, 2002 (gmt 0)


I have read the robots.txt tutorial and I'm still a bit
unclear about where the robots.xtx file belongs or where
to find it.

I do see that when a spider comes a crawlin' they seem
to find my robots.txt file just fine.

The question is: where the heck is that file? The tutorial
says to put it in the root folder...does that mean the
robots.txt file should be included in the html of the index
page?

P.S. I use dreamweaver 4 if that matters.


Thread source:: http://www.webmasterworld.com/robots_txt/135.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com