i have a folder password protected, this folder will contain all private webpages.
if i do nothing to keep spiders out, will they be able to crawl these pages?
stapel
7:05 pm on Oct 26, 2005 (gmt 0)
If access requires entering a username and password, and since the spider does not have such and could not enter it even if it did, I don't see this as being a problem.
Eliz.
JAB Creations
1:19 am on Nov 1, 2005 (gmt 0)
Use Firefox, get the useragent switcher, set the useragent to that of any given spider, and then access the file in question.
It should get a 403 (just as Firefox did when I first asked myself this question).
