incywincy
msg:1528295 | 9:47 am on Jul 21, 2004 (gmt 0) |
What appears in your logs may just be an attempt by a spider to retrieve this file, it doesn't necessarily mean that the file exists. What was the HTTP response code returned by your webserver?
|
uk_webber
msg:1528296 | 2:38 pm on Jul 21, 2004 (gmt 0) |
404! Of course. Thanks for pointing that out. I can rest now! Any ideas why google will only index my first page?
|
elgumbo
msg:1528297 | 1:48 pm on Aug 2, 2004 (gmt 0) |
You should really add a robots file otherwise you will find your error logs get bunged down with 404 for it. As far as google not spidering your site, have you got any external links to your content pages (as oppposed to just the home page)? I've found this can certainly help.
|
broniusm
msg:1528298 | 2:59 pm on Aug 5, 2004 (gmt 0) |
Is there any risk to turning off a web crawler [webcrawler.com] (wow, it's still online!) with adding a robots.txt? (like, might they not get a 404 and just assume they're not allowed?) What do you put in a robots.txt that allows full site indexing?
|
|
