The best way to add uncrawlable links?

Following on from a post about spider friendly JavaScript for collapsible c

[jonny0000]

Following on from a post about spider friendly JavaScript for collapsible content, I wondered if anybody had intentionally used uncrawlable pieces of JA or Ajax to prevent search engines crawling specific links.

As an example, many ecommerce solutions have multiple product links which screws up the balance of the internal linking structure. If you could restrict access to duplicate links and allow bots to crewel only a single text link (whilst not impacting usability) then the internal linking structure would be a lot healthier.

For the greater good make certain areas uncrawelable. Thoughts?

[goodroi]

#1 It depends on the situation.

#2 For some situations it can definitely help to block the spiders from reaching certain links & pages. I have done it on a few projects and have been rewarded with better rankings & traffic.

[dstiles]

All my web sites (except where a customer requests otherwise) block or remove links from certain pages (eg contact forms) if a bot of any kind is detected, in addition to the page being blocked in robots.txt. If the bot hits the unlinked page then it gets a 405 returned.

[jonny0000]

Thanks dstiles. I am thinking about preventing access through certain channels rather than preventing access to certain pages altogether where of course robots protocol could be used.

goodroi, what methods have you tested when preventing access through certain links and which have you seen the best results with?

[jameswsparker]

You can request Google to crawl specific sites/links for it's search engine by adding the URL into Google:

[google.co.uk...]

Entering your site here will get Google to crawl all the links it can find on that site.

[goodroi]

every situation is different. i have personally used robots.txt, metarobots and a few others. the method i use depends on the scale of the project and the preexisting structure.