Welcome to WebmasterWorld Guest from 22.214.171.124
Forum Moderators: goodroi
Here is the robots.txt file:
#Robots.txt for www.MyDomain.com
#Email editor@MyDomain.com with any questions.
I am trying to disallow spiders from indexing the /images folder and the Purchase.php page. Am I doing more than I'm intending?
I do find that when I follow a link at www.MyDomain.com, I end up dropping the "www." and go to MyDomain.com/MyPage.php.
Would that mess up a spider? I'm using relative links ... /MyDirectory/MyPage.php ... throughout.
Thanks for the help!
There doesn't look to be anything on your index page to stop them going further. You might try putting up a fresh link or two on the page and see if they follow those.
"I took a look at your account and I noticed that your page has links such as
[MyDomain.com...] Because there is a slash missing from
the end to make the URL [MyDomain.com...] the spider
receiveds a re-direct to [MyDomain.com...] but cannot
follow the re-direct. "
That was the case for one spider, at least ... possibly more.
Hope this helps somebody else -- and thanks to those who gave me some ideas earlier.