jimbeetle
msg:1527647 | 7:02 pm on Dec 11, 2003 (gmt 0) |
I can't see anything that should stop them. Your file validates in Brett's robots.txt validator [searchengineworld.com]. The only thing might be to insert a space between the # and actual comment, not sure how nitpicky some spiders are, but that's the format used at robotstxt.org. There doesn't look to be anything on your index page to stop them going further. You might try putting up a fresh link or two on the page and see if they follow those.
|
dwilson
msg:1527648 | 7:10 pm on Dec 11, 2003 (gmt 0) |
Thanks, Jim. I do have some more content almost ready to add, so I'll provide a link from the main page & see if they get it.
|
engine
msg:1527649 | 7:18 pm on Dec 11, 2003 (gmt 0) |
>#Email editor@MyDomain.com with any questions. I'd remove that to help cut back on e-mail spam.
|
dwilson
msg:1527650 | 7:20 pm on Dec 11, 2003 (gmt 0) |
Thanks, Engine. Good idea.
|
dwilson
msg:1527651 | 8:55 pm on Dec 11, 2003 (gmt 0) |
A service I was trying to use for my site-level search (not G, as I can't get them to re-index whenever I want) explained my problem. "I took a look at your account and I noticed that your page has links such as
[MyDomain.com...] Because there is a slash missing from
the end to make the URL [MyDomain.com...] the spider
receiveds a re-direct to [MyDomain.com...] but cannot
follow the re-direct. " That was the case for one spider, at least ... possibly more. Hope this helps somebody else -- and thanks to those who gave me some ideas earlier.
|
|
