homepage Welcome to WebmasterWorld Guest from 54.197.183.230
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
robots.txt and IP addresses
Maciej Ziemczonek

5+ Year Member



 
Msg#: 3621625 posted 10:17 am on Apr 8, 2008 (gmt 0)

Hi

I have a problem. Our site is located on a couple of servers. Google is somehowe indexing also direct urls of these servers.

Therefore, for example, we have in google:

www.ourdomain.com

and

www.ourdomain.hostnameserver1.com
www.ourdomain.hostnameserver2.com

What's more, also IP addresses of these severs are indexed as seperate urls.

We have only one robots.txt, the same meta tags are for the whole site. The structure of catalogues and files is the same for each server - which makes it impossible to block access to chosen catalogues and files.

Do you have any idea how to:

1. block Googlebot from crawling these urls?
2. remove these urls from Google index?

I'd appreciate you help greatly,

best regards
Maciej

 

goodroi

WebmasterWorld Administrator goodroi us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 3621625 posted 1:03 pm on Apr 9, 2008 (gmt 0)

hello Maciej,

it sounds like whoever setup your hosting created mirrors of your site at multiple locations. robots.txt will not help this situation since the robots.txt file will be copied to all the duplicate places. you should talk to your hosting person and explain that only one url should be indexed. they can make some changes to the hosting setup to deal with this situation.

as for duplicate content you should not have too big of an issue as long as you have all of your link popularity pointing to one version of the site. the engines will filter out the other duplicates.

good luck

Maciej Ziemczonek

5+ Year Member



 
Msg#: 3621625 posted 1:39 pm on Apr 9, 2008 (gmt 0)

Goodroi - thanks for your help!

g1smd

WebmasterWorld Senior Member g1smd us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3621625 posted 12:01 am on Apr 10, 2008 (gmt 0)

If you simply stop the alternative URLs working, you will lose that traffic.

You need a site-wide 301 redirect to fix this.

The redirect will preserve the traffic that comes in through the wrong URLs.

The 301 redirect will ensure that the wrong URLs are eventually de-indexed.

Ensure that all your internal linking points to the canonical domain.

Maciej Ziemczonek

5+ Year Member



 
Msg#: 3621625 posted 6:39 am on Apr 10, 2008 (gmt 0)

Thanks!

vietbds

5+ Year Member



 
Msg#: 3621625 posted 9:54 am on Apr 11, 2008 (gmt 0)

good infomation.
Thanks

garryb

5+ Year Member



 
Msg#: 3621625 posted 8:52 am on Apr 16, 2008 (gmt 0)

I yesterday put a redirect from http://example.ie to http://www.example.ie as both were appearing in the search engines. A friend told me that these two sites, although the same will be competeing with each other and search engines might see them as duplicates.
I put a rewrite rule in my .htaccess file. Does anyone know if this will cause prolems?

[edited by: goodroi at 1:02 pm (utc) on April 16, 2008]
[edit reason] Examplified [/edit]

goodroi

WebmasterWorld Administrator goodroi us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 3621625 posted 1:15 pm on Apr 16, 2008 (gmt 0)

hi garryb,

welcome to webmasterworld. having the content accessible at both URLs can cause issues in the search engines. it is ideal to use a 301 redirect and point one of them into the other (which you have done). by redirecting you make sure the link popularity is focused and not divided. this also minimizes issues with being flagged as duplicate content.

g1smd

WebmasterWorld Senior Member g1smd us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3621625 posted 6:32 pm on Apr 16, 2008 (gmt 0)

The 301 redirect needs to be site-wide, not just for the root.

bilalseo

5+ Year Member



 
Msg#: 3621625 posted 10:10 pm on May 2, 2008 (gmt 0)

agree with g1smd ;)

bilalseo

5+ Year Member



 
Msg#: 3621625 posted 10:11 pm on May 2, 2008 (gmt 0)

agree with g1smd ;)

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved