homepage Welcome to WebmasterWorld Guest from 54.205.254.108
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Would this work? Robots exclusion or considered cloaking?
dvldvl3




msg:1527789
 9:36 am on Jul 8, 2005 (gmt 0)

hi,

we have a travel site which has about a thousand pages, and has been online in the search engines for about 5 years. it uses a CMS, one that outpts in html.

we have never done any real seo, apart from changing a few tags around since last year. however it ranks ok for the keywords we have chosen.

we are going to do a total re-design. the new design proposed has a lot of rich media content, is still using the cont tent management system, and will be very graphic oriented.

if we made a duplicate version for accessibility reasons , in plain html, exactly like the bbc have done . (u can see this under 'text only' version on the bbc main page - bbc.co.uk/home/today/textonly.shtml) and then put a robot exclusion tags on every page of the rich media site. could this work?

so in effect we have a index splash page. 2 entry points on that page. main visitors will go to the colourful graphic pages. that link blocks the search engines. they follow the links to the text only pages. previous link popularity still going to main index page, + new links deeplinking to text only pages.

is this considered a kind of cloaking? or is it just an alternative way of doing things. we would have to have the accessability part anyway for legal reasons as it is a brand site. this way would just be taking advantage of seo at the same time.

we would also need to have the robot exclusion tags because of the duplicate content. this way there would only be 1 set of copy indexed and it would just be the easy to spider text stuff.

any help, info or ideas would be appreciated thanks.

 

Abdelrhman Fahmy




msg:1527790
 1:38 am on Jul 10, 2005 (gmt 0)

i don't think that is a duplicate by any way as its an added value to your site visitors ,
also you could put all of the plain text Content in a separate folder and block the access to the whole folder by Robots.txt

wattsnew




msg:1527791
 3:41 am on Jul 10, 2005 (gmt 0)

I'd like to hear everyone's input on this one too.

Logically, this should work as you are not attempting to manipulate the SEs with the graphical pages and if blocked, they should not appear to be "duplicates" of the text pages.

However, many pages I have blocked with NOINDEX, NOFOLLOW turn up as URL only listings. In a couple of cases, where they were, in fact, near duplicates of an indexed page, the near duplicate indexed page has been dropped to URL only as well. Not the plan at all! Would like to know why.

Where I have put near duplicate pages in a separate directory (eg landing pages for AdWords), the directory blocked entirely by robots.txt, it seems to have been 99% effective: no sign of these in Google (except where a stray link may have crept in) and no related duplicate penalties.

ThomasB




msg:1527792
 7:56 am on Jul 14, 2005 (gmt 0)

dvldvl3, first of all Welcome to the robots.txt Forum at WebmasterWorld!

I'd definetly try to avoid having both pages indexed as this could cause problems with the search engines. I'd put, as already suggested, both versions in separate folders and disallow one of them via the robots.txt-Standard. This makes handling very easy as you don't have to adjust the robots.txt file every time you add/change/remove a file that's in the root directory.

Reid




msg:1527793
 5:11 am on Jul 17, 2005 (gmt 0)

another way to accomplish this is with the same page but different .css files.
Definitely allow the text-based low-graphics page to be spidered.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved