Forum Moderators: goodroi
In an attempt to stop this, I have replaced all outbound links as follows:
www.domain.com/subpage
becomes
/go.php?url=www.domain.com/subpage
I have created the following robots.txt:
User-agent: *
Disallow: /go.php
After all this, google is passing PR and listing my site as backlinks for the recipients.
Have I done something wrong or do I need to encrypt the URLs?
Best regards
Shady
A few months ago I noticed that google was spidering many instances of a form that is on my site e.g. Goggle had form.asp?id=2 form.asp?id=3 etc.
My robots.txt now reads:
User-agent: *
Disallow: form.asp
However more than 6 months later (and Google has been back and re-cached the page several times), Google is still including the page.
A clarification from Google would be nice.
After all this, google is passing PR and listing my site as backlinks for the recipients.
The purpose of a robots.txt file is to ask robots to not download files.
Was your "go.php" page actually downloaded and indexed by Google? If not, Google has followed your robots.txt directives.
Disallow: form.asp
That should be "Disallow: /form.asp".
Try to analyze your robots.txt files with a robots.txt validator.
Thanks for you advice regarding the use of "/" before the filename.
Actually many of the "authority" sites on Robots.txt do not specify the need for a preceeding forward slash and my robots.txt file seems to validate okay without it.
Anyway, I have updated my robots.txt with your suggestion, so hopefully the "/" will do the trick.
Here is my current Robots.txt file info:
User-agent: *
Disallow: /cgi-bin/
Should I specify the Googlebot as well? Thanks!
