homepage Welcome to WebmasterWorld Guest from 54.161.191.254
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
robot.txt restricted but still showing !
member22




msg:3761275
 2:30 pm on Oct 8, 2008 (gmt 0)

Hey guys,

I think I am shooting my self in the foot with the robot.txt file I have on my website

Here is my question :

I use to have a webpage that I created in my back office named www.mywebpage.com/p123/mywebpage.html

I restricted this page with my robot that.txt file because it doesn't exist anymore
and my new new page address is www.mywebpage.com/p234/mynewwebpage.html ( the way my website is built the .html name doesn't matter only the number matters ) !

but I noticed that when I type the old address that I thought was restricted it brings me to my current page and my webpage has 2 different adress for the same page with 2 different numbers ?

www.mywebpage.com/p123/mywebpage.html
www.mywebpage.com/p234/mynewwebpage.htm

I don't understand why ? what should I do should still restrict the old page or should I not restrict anything and let google figure out the right page ?

Thank you,

 

Pitafi




msg:3761286
 2:42 pm on Oct 8, 2008 (gmt 0)

Hi,

i think that you should restrict your old page, so that google may not figure it out.if google figured it out then it will create some problems, like google may ban your site or apply some penalty.

So far i have visited a site that has two url's of same page, but google still showing on the top position in SERP's.

thanks
ammar

member22




msg:3761335
 3:36 pm on Oct 8, 2008 (gmt 0)

Thank you for the reply but my second question is what should I do if I have an old page with the same number but different .html should I still restrict the old page

Ex :
my old page is [myoldpage.com...]

and my new page is

[myoldpage.com...]

The only thing that changes is the .html but in my case my what only counts on my website is the number P123

My guess is I shouldn't restrict anything and just let google do his thing and change the .html title when it finds the new one don't you think ?

Pitafi




msg:3761408
 4:59 pm on Oct 8, 2008 (gmt 0)

For google, content should not be the same. if you have two url's with two different numbers showing the same content, this will hurt your site rank, coz google will consider it as duplicate content.

ammar

member22




msg:3761412
 5:01 pm on Oct 8, 2008 (gmt 0)

so, restricting one of the url ( problably the old one ) with duplicate content is the solution isn't it ?

Pitafi




msg:3761419
 5:07 pm on Oct 8, 2008 (gmt 0)

i think you should apply 301 redirect on old page to your new page.
doing this content will not be the issue.

Pitafi




msg:3761421
 5:08 pm on Oct 8, 2008 (gmt 0)

i think you should apply 301 redirect on old page to your new page.
doing this content will not be the issue.

g1smd




msg:3770365
 1:25 pm on Oct 21, 2008 (gmt 0)

If a page has moved, the a 301 redirect from old to new is required.

The redirect should work whether the old URL was requested as www or as non-www, and the redirect should force the new URL to have the www in it... otherwise that is another form of Duplicate Content.

jdMorgan




msg:3770406
 2:05 pm on Oct 21, 2008 (gmt 0)

Disallows are based on URLs, not on filenames or the content served for those URLs. So disallowing the old URL will not affect the new URL, as long as your Diallow directive specifies the full path to the old URL, and the new URL's prefix does not match the Disallowed URL.

However, as pointed out above, the proper approach is to 301-redirect the old URL to the new URL, and robots.txt should not be involved.

After this primary issue is resolved, do take a look around here (using the WebmasterWorld site search) for threads on "duplicate content." Your statement that "in my case my what only counts on my website is the number P123" indicates that the same content will be returned for http://www.example.com/p123/<anything>, which is a recipe for a duplicate-content disaster, especially if one of your competitors discovers this vulnerability and exploits it to create a bunch of non-canonical links to your pages in order to destroy the ranking of your canonical URLs...

Jim

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved