Forum Moderators: goodroi
I've disallowed dynamic urls in my current robots.txt file so these supplemental results pages should *eventually* disappear - but who knows when?
If I use the Google URL removal tool with my current robots.txt file it would probably temporarily remove the duplicate pages from the visible index for 180 days. Yes, I can't use the google disallow wildcard with the google url removal tool, but dissallowing "/cgi-local/" on my site would also remove dynamic urls. However, if they are going to pop back up six months later in the supplemental results again, using the URL removal tool seems like a waste of time.
Am I correct that it would be useless to use the URL removal tool to try to get rid of these supplemental results, and that I might as well just wait it out and count on my robots.txt file to eventually get rid of these duplicate pages?
On the plus side, Googlebot is no longer requesting them. My webmaster tools shows these pages all dynamic pages as "blocked".
Any feedback on this issue greatly appreciated.
Although I could use robots.txt to remove the duplicate pages, my gut feel is that it isn't going to make any difference in terms of PR, pages spidered or anything else. I assume the pages will eventually disappear because robots.txt is blocking them.
I was confused in my first post because I didn't see Googlebot spidering them in my log files. Actually, Google Webmaster Tools shows the page requests being blocked by robots.txt. GWT's shows 16,000 plus blocked page requests - all for the duplicate sessionid data pages.
Although I am confident of robots.txt file as a removal tool, if something went wrong and it took site completely out of index it would be curtains for us. We are an ecommerce site doing nearly 7 figures. Pretty risky tactic no matter how carefully I use URL removal tool, right?
[edited by: Tonerman at 4:22 pm (utc) on Oct. 30, 2006]