Welcome to WebmasterWorld Guest from 126.96.36.199 , register , free tools , login , search , pro membership , help , library , announcements , recent posts , open posts Become a Pro Member
how to prevent directory from being indexed? dan_popescu msg:703141 9:57 pm on Dec 2, 2002 (gmt 0) I just made a robots.txt file for my site. I've included a few files that I don't want indexed. How do I include an entire directory in robots.txt? I know the syntax for files is "disallow: file.htm". Is the syntax for an entire directory (and its content) the same: "disallow: directory"?
Quinn msg:703142 9:59 pm on Dec 2, 2002 (gmt 0)
Disallow: /directory/ Quinn msg:703143 9:59 pm on Dec 2, 2002 (gmt 0)
and this should disallow specified bots from everything contained in that folder... dan_popescu msg:703144 10:04 pm on Dec 2, 2002 (gmt 0)