Welcome to WebmasterWorld Guest from 220.127.116.11
Forum Moderators: goodroi
I did check here just to be sure
implies that files will be matched (though it does not explicitly say that folders will be matched.)
Remember to add the trailing slash ("/") if you are indicating a directory. If you simply add
the robots will be disallowed from accessing privatedata.html as well as privatedataandstuff.html as well as the directory tree beginning from /privatedata/ (and so on). In other words, there is an implied wildcard character following whatever you list in the Disallow line.
I have always done all files and directories explicitly so I have never worried about it.
Me neither, but since robots.txt is human readable, its a good way to tell hacker kiddies what folders you dont want them in.
Thanks for replying!
--Kenn (and then there is the argument about using * in the disallow... some places say yes, some say no....)