Forum Moderators: goodroi

Message Too Old, No Replies

Google remove url tool for supplemental urls a waste of time?

Should I bother removing duplicate supplementals?

         

Tonerman

7:34 pm on Oct 28, 2006 (gmt 0)

10+ Year Member



My site has a little over 300 pages. Google's site command shows 2090 pages, or roughly 1700 duplicate pages with different session id data.

I've disallowed dynamic urls in my current robots.txt file so these supplemental results pages should *eventually* disappear - but who knows when?

If I use the Google URL removal tool with my current robots.txt file it would probably temporarily remove the duplicate pages from the visible index for 180 days. Yes, I can't use the google disallow wildcard with the google url removal tool, but dissallowing "/cgi-local/" on my site would also remove dynamic urls. However, if they are going to pop back up six months later in the supplemental results again, using the URL removal tool seems like a waste of time.

Am I correct that it would be useless to use the URL removal tool to try to get rid of these supplemental results, and that I might as well just wait it out and count on my robots.txt file to eventually get rid of these duplicate pages?

On the plus side, Googlebot is no longer requesting them. My webmaster tools shows these pages all dynamic pages as "blocked".

Any feedback on this issue greatly appreciated.

goodroi

2:25 pm on Oct 30, 2006 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



If Googlebot is no longer requesting them and you have correctly blocked them in your robots.txt file then based on my experience you should be ok.

Side Note - Always think twice before using the Google removal tool since it lasts for 6 months and there is no way of undoing it.

Tonerman

4:20 pm on Oct 30, 2006 (gmt 0)

10+ Year Member



I truly appreciate your feedback! I mis-spoke when I said no longer requesting. In fact, my google webmaster tools reports show Google is requesting them but robots.txt is disallowing the pages because they have "?" in the URLS.

Although I could use robots.txt to remove the duplicate pages, my gut feel is that it isn't going to make any difference in terms of PR, pages spidered or anything else. I assume the pages will eventually disappear because robots.txt is blocking them.

I was confused in my first post because I didn't see Googlebot spidering them in my log files. Actually, Google Webmaster Tools shows the page requests being blocked by robots.txt. GWT's shows 16,000 plus blocked page requests - all for the duplicate sessionid data pages.

Although I am confident of robots.txt file as a removal tool, if something went wrong and it took site completely out of index it would be curtains for us. We are an ecommerce site doing nearly 7 figures. Pretty risky tactic no matter how carefully I use URL removal tool, right?

[edited by: Tonerman at 4:22 pm (utc) on Oct. 30, 2006]