Forum Moderators: open
Our website writes all URL's found throughout the site as static URLs. There are some dynamic URL's but I use both robots.txt and rel="nofollow" on all of those links.
But, we run our own affiliate program and in order to track their clicks we must let them us dynamic URL's.
The problem is... the SE's are using some of our URL's as the dynamic affiliate URL instead of our static URL. I don't understand the logic behind this one. As every page on our site points to the static URL. That, and our robots.txt file clearly says to ignore the dynamic URL.
So my question is... is there anything I can do to stop the SE's from indexing the dynamic URLs? Our static URL's rank A LOT better and we've lost a lot of high ranking positions because of the new dynamic URLs.
Any help would really be appreciated.
...the SE's are using some of our URL's as the dynamic affiliate URL instead of our static URL
midoriweb - Can you clarify what you mean by "dynamic affiliate URL"?
Also, what's the inbound linking situation? Is anyone linking to your dynamic urls?
Dynamic Affiliate URL's are urls that are dynamic and include the tag at the end to track each affiliate (for example: &Click=1234).
An affiliate url would like like:
mysite.com/ProductDetails.asp?ProductCode=product1&click=1234
The actual URL I have on my site that I want spidered is:
mysite.com/Product_Name/productcode1.htm
Problem is... Yahoo, and now a little of google is indexing the affiliate URL instead of the URL located on my website.
So to answer your second question, yes, people are linking to the dynamic URL's. But I don't want those URL's showing in the SE's. I only want my static URL's being listed.
This appears to be a situation where the urls of the inbound links with various tracking strings have caused the engines to index the page under multiple urls.
A page's content will be indexed under as many urls as there are different inbound tracking strings. These multiple urls are seen as dupe content by the engines, and the engines sort out dupes in various ways. Generally, on Google at least, the url with the highest PageRank wins.
Usually, a single affiliate link contributes only a small fraction of the PageRank of a page, so affiliate links generally don't show up in the serps. Yahoo seems to have more problems with tracking strings than the other engines do... but generally Yahoo also sorts this out.
A high PageRank inbound affiliate link, though, can be a source of your problem. You're also likely to have problems if you have no other external inbounds to the "correct" url of your page (ie, to what you would call your "static" url).
I'd recommend getting other non-affiliate inbound links to your affected pages, and/or to the site as a whole.
Thank you for the reply. We're in the process of trying to get ISAPI_Rewrite to modify our URL's to allow affiliates to point to our static HTML pages and still record their affiliate tracking.
I guess some SE's will determine that if 200 sites point to a certain URL for a product but our site only links to a different URL, the one that people link to is most relevant and will be shown in the results. That makes sense.
I just wish SE's would follow our robots.txt file and NOT index the pages we have listed there. But I'm guessing robots.txt is only for spiders to be told where to go and where not to go. When it comes to indexing items in SE's, if people link to pages you have as dissallowed in your robots.txt file they will still show up in the search results.
Here's the twist... If you do that, but also use robots.txt to keep the page itself from getting crawled, Googlebot won't crawl the page and therefore won't see the robots "noindex" meta tag, so the links will get indexed.
If Request.ServerVariables("url") = "/ProductDetails.asp" Then
wl("<meta name=""robots"" content=""noindex,follow"">")
wl("<meta name=""googlebot"" content=""noindex,follow"">")
Else
wl("<meta name=""robots"" content=""index,follow"">")
wl("<meta name=""googlebot"" content=""index,follow"">")
End If
That way... if the ProductDetails.asp page is called robots will be told not to index. But if any other page is called (like my SE friendly URL's, it will index. This is the best way to get around my situation?
So the best thing would be to include all directories or files you don't want indexed in your robots.txt file. But... if people are going to link to certain directories or files you don't want indexed, then remove them from your robots.txt file and include the noindex metatag instead?
Your question is too general for me to answer. I still don't know, eg, whether you have dynamic pages that get dynamic affiliate links, and then you have static versions of these pages for general searchers to find.
If you're set up this way, you definitely wouldn't want to be blocking your static pages... but you might be able to set up all of your dynamic pages to use the meta noindex tag.
On the other hand, if all of your pages are static and you're talking about "dynamic" links to them, I'd think you'd run into SEO problems blocking them at all.
I do not have a clue how you'd manage a situation where affiliates might link to some pages and not others without also blocking search engine access. And you might need a full time person just to manage your robots.txt. ;)
Something I'd pursue in the robots.txt forum.
Our ProductDetails.asp page is dynamic and only linked to by affiliates. Our site uses ProductDetails.asp to display our static URL's as well (our store software will convert the static URL and process it with ProductDetails.asp (if that makes sense).
So... both pages show the exact same content... it's just the URL that's different. I made it so if the page is called from ProductDetails.asp it displays the Meta noindex tag (this will only display when affiliates link to the site). If the page is called by the static URL as shown throughout our site noindex will not be displayed in the headers.
Thank you for clarifing everything though. If you didn't explain the catch-22 I would have modified the metatag noindex and I would have left the robots.txt the same.
So... both pages show the exact same content... it's just the URL that's different.
midoriweb - In that case, I would dynamically insert the robots noindex meta into all of the dynamic pages... make sure it's not also on the static pages... and not use robots.txt to block the dynamic pages at all.
But new question... I can do what xxcli#*$! said before your post. I can put a 301 redirect on my ProductDetails.asp page. So now, if a user views the dynamic page they will be redirected to the static page (and their code will still be still be tracked).
But... what does this do as far as links in the SE?
If someone links to the ProductDetails.asp page (dynamic) will the SE's pick up the redirection and list the page as the static URL? Or... will the SE's still continue to list the URL as a dynamic URL?
The plus side to doing this redirection is that all links and PR should go towards one page instead of multiple URL's for 1 single page. I'm thinking right now that a 301 redirect of dynamic URL's to a static URL is the best way to optimize my site. My only concern would be how SE's are dealing with my redirection. Will they still displays incoming dynamic links in the search results... or will they update their dynamic incoming links and start to display my static URL's?
If one takes the approach that a 301 is a 301, then the referring url should be rewritten, and should not be indexed. On the other hand, this might be the kind of 301 that people are complaining about staying on in the engines.
I think you're going to need someone who knows servers better than I do for this answer.
I'll try posting this question in another forum to see what people think.
