Welcome to WebmasterWorld Guest from 220.127.116.11
Forum Moderators: goodroi
implies that files will be matched (though it does not explicitly say that folders will be matched.)
Remember to add the trailing slash ("/") if you are indicating a directory. If you simply add
the robots will be disallowed from accessing privatedata.html as well as privatedataandstuff.html as well as the directory tree beginning from /privatedata/ (and so on). In other words, there is an implied wildcard character following whatever you list in the Disallow line.
I have always done all files and directories explicitly so I have never worried about it.
Me neither, but since robots.txt is human readable, its a good way to tell hacker kiddies what folders you dont want them in.
Thanks for replying!
--Kenn (and then there is the argument about using * in the disallow... some places say yes, some say no....)