Forum Moderators: coopster
Two questions:
1. Has this already affected my search engine ranking?
2. How can I restrict the crawlers from indexing this script (which has lots and lots of urls thanks to url rewriting)? I can't use robots meta tags since the script has no actual output, and a robots.txt might prove difficult... Can I restrict indexing from within the script?
And then just ban spiders from indexing /redir using robots.txt.
I'm not sure I understand the situation, so this answer could be way off :)
I had my hopes on an easier way. Maybe I just have to generate robots.txt. If not anyone has a great idea? I would prefer to stop the bots from crawling these URLs asap...