Forum Moderators: coopster

Message Too Old, No Replies

Redirecting and search engines

         

chrisjoha

4:05 pm on Oct 25, 2005 (gmt 0)

10+ Year Member



I have a script that takes some parameters and redirect using header(). So it has no actual output. So, now I discover that I forgot search engines, and today this script got crawled an insane amount of times...

Two questions:
1. Has this already affected my search engine ranking?
2. How can I restrict the crawlers from indexing this script (which has lots and lots of urls thanks to url rewriting)? I can't use robots meta tags since the script has no actual output, and a robots.txt might prove difficult... Can I restrict indexing from within the script?

mykel79

8:40 pm on Oct 25, 2005 (gmt 0)

10+ Year Member



What do the urls that serve up the script look like?
Maybe you could have all of them pretend to be in a subdirectory and ban the spiders from that directory?
e.g.
http://www.example.com/redir/blabla1.php
http://www.example.com/redir/blabla2.php
http://www.example.com/redir/blabla3.php

And then just ban spiders from indexing /redir using robots.txt.

I'm not sure I understand the situation, so this answer could be way off :)

chrisjoha

10:31 pm on Oct 25, 2005 (gmt 0)

10+ Year Member



Well, it is _possible_, but I was hoping I could work some magic from the script itself and don't have to worry about it anymore :) The reason is that the url-structure is dynamic, ie /<db-value>/<another-db-value>/<and-so-on>/ This means that I would either need to write a script to serve up robots.txt OR compromise my beautifully clean URLs :)

I had my hopes on an easier way. Maybe I just have to generate robots.txt. If not anyone has a great idea? I would prefer to stop the bots from crawling these URLs asap...