robot.txt restricted but still showing !

[member22]

Hey guys,

I think I am shooting my self in the foot with the robot.txt file I have on my website

Here is my question :

I use to have a webpage that I created in my back office named www.mywebpage.com/p123/mywebpage.html

I restricted this page with my robot that.txt file because it doesn't exist anymore
and my new new page address is www.mywebpage.com/p234/mynewwebpage.html ( the way my website is built the .html name doesn't matter only the number matters ) !

but I noticed that when I type the old address that I thought was restricted it brings me to my current page and my webpage has 2 different adress for the same page with 2 different numbers ?

www.mywebpage.com/p123/mywebpage.html
www.mywebpage.com/p234/mynewwebpage.htm

I don't understand why ? what should I do should still restrict the old page or should I not restrict anything and let google figure out the right page ?

Thank you,

[Pitafi]

Hi,

i think that you should restrict your old page, so that google may not figure it out.if google figured it out then it will create some problems, like google may ban your site or apply some penalty.

So far i have visited a site that has two url's of same page, but google still showing on the top position in SERP's.

thanks
ammar

[member22]

Thank you for the reply but my second question is what should I do if I have an old page with the same number but different .html should I still restrict the old page

Ex :
my old page is [myoldpage.com...]

and my new page is

[myoldpage.com...]

The only thing that changes is the .html but in my case my what only counts on my website is the number P123

My guess is I shouldn't restrict anything and just let google do his thing and change the .html title when it finds the new one don't you think ?

[Pitafi]

For google, content should not be the same. if you have two url's with two different numbers showing the same content, this will hurt your site rank, coz google will consider it as duplicate content.

ammar

[member22]

so, restricting one of the url ( problably the old one ) with duplicate content is the solution isn't it ?

[Pitafi]

i think you should apply 301 redirect on old page to your new page.
doing this content will not be the issue.

[Pitafi]

i think you should apply 301 redirect on old page to your new page.
doing this content will not be the issue.

[g1smd]

If a page has moved, the a 301 redirect from old to new is required.

The redirect should work whether the old URL was requested as www or as non-www, and the redirect should force the new URL to have the www in it... otherwise that is another form of Duplicate Content.

[jdMorgan]

Disallows are based on URLs, not on filenames or the content served for those URLs. So disallowing the old URL will not affect the new URL, as long as your Diallow directive specifies the full path to the old URL, and the new URL's prefix does not match the Disallowed URL.

However, as pointed out above, the proper approach is to 301-redirect the old URL to the new URL, and robots.txt should not be involved.

After this primary issue is resolved, do take a look around here (using the WebmasterWorld site search) for threads on "duplicate content." Your statement that "in my case my what only counts on my website is the number P123" indicates that the same content will be returned for http://www.example.com/p123/<anything>, which is a recipe for a duplicate-content disaster, especially if one of your competitors discovers this vulnerability and exploits it to create a bunch of non-canonical links to your pages in order to destroy the ranking of your canonical URLs...

Jim