| Testing robots.txt
Why won't my site get spidered |
DavidAtWork
msg:1526899 | 7:51 pm on Feb 4, 2004 (gmt 0) |
Hi,
I have started working with robots.txt to try and understand what is being spidered and what I need to filter, etc.
In testing I noticed (using the spider simulator on search engine world) that my site refuses the spider. (Even if I don't have a robots.txt) I get a 403 denied error and it says that I have to allow directory browsing. I have other sites that I don't have directory browsing allowed on and they get spidered and I looked at the folder permissions and they are the same. Even the iis properties are the same. Just to test it. I checked off allow directory browsing and it worked. But obviously I don't want to leave it like that. Any suggestions as to what might be the issue? thanks.....
|
closed
msg:1526900 | 5:56 pm on Feb 5, 2004 (gmt 0) |
Maybe your directory doesn't have a default page (like index.html)?
|
|
|
