Welcome to WebmasterWorld Guest from 23.20.13.165

Forum Moderators: Robert Charlton & andy langton & goodroi

Message Too Old, No Replies

How to stop specific sections of a page from being crawled?

     
9:01 am on Aug 4, 2015 (gmt 0)

New User

5+ Year Member

joined:June 9, 2008
posts: 23
votes: 0


I would like to display customer reviews from another website (with their agreement) own my own site for certain product pages.

Using these 3rd party reviews would be helpful for users, though there's an issue with duplicate content as I'll be republishing these reviews from the other website.

I don't want to noindex the whole page as many of these product pages already rank well.

One option I'm considering is to place the review content in an iframe and then block the iframe's URL in the robots.txt, though I'm unclear as to whether this would be frowned upon by search engines.

Is there a best practice, recognized and accepted by Google, for noindexing or blocking crawling of only certain sections of a page?
8:06 pm on Aug 4, 2015 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member netmeg is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Mar 30, 2005
posts:13002
votes: 212


First of all, you can block content in robots.txt but that won't control indexing, and if there are links to that content, it'll just go in without a meta description.

Is there a best practice, recognized and accepted by Google, for noindexing or blocking crawling of only certain sections of a page?


I seem to recall not. I mean, it would be open to a ton of spam and abuse if they did. More than normal, I mean.
8:32 pm on Aug 4, 2015 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member themadscientist is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 14, 2008
posts:2910
votes: 62


I think one of the only ways you *might* get what you want is to:

1.) Put the comments on a separate page.
2.) Noindex the page their on.
3.) Put a link to the specific comments on the product page their related to.
4.) Nofollow the link to the page.
5.) Attach a click function to the link (or text that looks like a like but isn't really a link) from the product page to the comments page after the page renders [window.onload = function() | $(document).ready(function() {});] that cancels the default action of clicking the link taking the visitor to the comments page and instead adds the comments to the current product page via ajax.

Not sure I'd recommend doing it and I haven't tested it, but it's one of the only ways I can think of to *maybe* have the comments separate from the content of the products page.
7:26 pm on Aug 5, 2015 (gmt 0)

Senior Member

WebmasterWorld Senior Member bwnbwn is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Oct 25, 2005
posts:3547
votes: 19


putting it in an iframe since it will be a separate page will work just add meta tags to no index, no archive, no follow, no anything. You can even add a no follow on the iframe url as well.
8:09 pm on Aug 5, 2015 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member themadscientist is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 14, 2008
posts:2910
votes: 62


Google's been associating iFramed content with the content of the containing page for quite some time, and with the known use of Chrome data, displaying the content on the page without a user-interaction to do so means they will not only very likely "discover the content" even if the iFramed link is nofollowed, they will very likely [almost certainly, imo] associate it with the content of the page.

What I suggested is much closer to "changing the page state" and displaying the content the user elected to see, by taking an action on the page, rather making the user wait for a full new page to load to show them the information they requested -- I'm still not sure my suggestion will even work but I suggested it as an alternative to an iFrame, because it's almost guaranteed an iFrame will not dissociate the content of the iFrame from the page containing the iFrame, since Google's been associating iFramed content with the page containing the iFrame for years, but a "user action to show the content from another location" *might* not be considered part of the page.

The "iFrame Trick" worked well in 2008ish, but not so much any more.
10:58 pm on Aug 5, 2015 (gmt 0)

Senior Member

WebmasterWorld Senior Member aristotle is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Aug 4, 2008
posts:3230
votes: 229


You can make images that show the text you want to display.
12:24 pm on Aug 6, 2015 (gmt 0)

Moderator This Forum from US 

WebmasterWorld Administrator robert_charlton is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 11, 2000
posts:11902
votes: 294


Here's one of several references that could be posted regarding Google's expressed intention to bring iframe and user experience in line with each other. Note Matt Cutts' commenta....

Iframe Links: Do They Pass Page Rank?
March, 2011
http://www.webmasterworld.com/google/4282550.htm [webmasterworld.com]

Eric Enge: If someone did choose to do that (JavaScript encoded links or use an iFrame), would that be viewed as a spammy activity or just potentially a waste of their time?

Matt Cutts: ...In my experience, we typically want our bots to be seen on the same pages and basically traveling in the same direction as search engine users. I could imagine down the road if iFrames or weird JavaScript got to be so pervasive that it would affect the search quality experience, we might make changes on how PageRank would flow through those types of links.

It's not that we think of them as spammy necessarily, so much as we want the links and the pages that search engines find to be in the same neighborhood and of the same quality as the links and pages that users will find when they visit the site.

The current discussion is regarding content, not links, but I think the underlying issues are the same. Clearly TheMadScientist is aware of these issues as well... thus his emphasis on "might" etc etc. Use of iframes is a technique where we've been assuming its days would be numbered. Ditto with javascript writes.

While I'm guessing that Google might eventually also use OCR to read text content in graphics, I have successfully used graphics, as aristotle suggests, for some boilerplate or headlines... occasionally for a paragraph... but never tried it for extensive comments. Seems to me that would be pushing things.

I've seen comments referenced simply by posting an excerpt only on the product page and then linking to a separate comments page as TMS suggests for the full set of comments. The question I would ask is whether all the ajax subterfuge gains you anything, or whether it might demonstrate deceptive intent and ends up hurting. It might be cleaner simply to link to a noindexed page and then identify these comments as "comments from elsewhere on the web" or some such.

I haven't tried this, but Google in its recent enhanced shopping results has shown some kind of acceptance for shared reviews of a product....

Searching a product shows reviews, what others are saying, specs, etc
https://www.webmasterworld.com/google/4756131.htm [webmasterworld.com]

Admittedly, Google is not constrained by dupe content requirements of organic search in these paid ads, so we need to be cautious in assuming it's OK... but there is an aspect of collective opinion about a product that seems to be acknowledged here by Google. I think that the acknowledgement of "what others are saying" would be an important part of this... along of course with the permission. I'm assuming the noindex would be necessary.
8:25 am on Aug 9, 2015 (gmt 0)

Full Member

10+ Year Member Top Contributors Of The Month

joined:June 3, 2005
posts:298
votes: 12


Horrendous using an iFrame.

Pull the content into the DOM using something simple like JSON and then put the JS code into a noindex folder.

jQuery can do this. Your JS is in the noindex folder but jQuery core itself does not have to be.

I presume AngularJS and KnockoutJS should also do this.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members