Welcome to WebmasterWorld Guest from 54.205.119.93

Message Too Old, No Replies

301 redirects and robots.txt

   
12:13 am on Jan 10, 2010 (gmt 0)

5+ Year Member



Our site has changed it's URL structure from www.example.com/pc to www.example.com/store/pc.

I know it's not ideal to add another category, but in order to be able to easily upgrade our cart system we need to keep their default URL structure. We did major mods to remove the store directory in the old system, and never really did get everything working right.

So I 301 redirected the www.example.com/pc pages to the new www.example.com/racing-parts/pc structure. It appears Google ranks all the old urls higher than the new ones, I suspect due to the extra directory away from the root.

So when should I add a dis-allow the www.example.com/pc to the robots.txt to get rid of the old URL's.

Any other thoughts on how to get Google to rank the new pages over the old ones?

We have a sitemap, etc...

1:42 am on Jan 10, 2010 (gmt 0)

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member



You should never need to add it. As Google finds the 301 redirects those URLs will naturally drop from the index.

Google has been a bit chaotic in recent weeks, doing some back-end changes to prepare for their new infrastructure. So lots of general spidering and housekeeping is behind schedule. As long as the visitor who clicks on the old URL is getting 301 redirected, you have no worries.

4:50 am on Jan 10, 2010 (gmt 0)

5+ Year Member



Yes, I have also tried 301 redirect. I needed to change the file name. So I set 301 to new file name. Until Google crawled my old URL again, I was getting copied content warning message in WMT. But as soon as my old URL was crawled again, the old URL was dropped from the index and also the warning message in WMT disappeared. So wait until your old URLs crawled again.
6:51 am on Jan 10, 2010 (gmt 0)

WebmasterWorld Administrator robert_charlton is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



So when should I add a dis-allow the www.example.com/pc to the robots.txt to get rid of the old URL's.

If you had added the disallow for the old urls, in fact, Google most likely would have been slower about dropping them, as Googlebot wouldn't have been able to crawl them and discover the 301s.

 

Featured Threads

Hot Threads This Week

Hot Threads This Month