Welcome to WebmasterWorld Guest from 54.159.50.111

Forum Moderators: goodroi

Message Too Old, No Replies

The best way to add uncrawlable links?

Following on from a post about spider friendly JavaScript for collapsible c

     
3:34 pm on Mar 16, 2010 (gmt 0)

New User

5+ Year Member

joined:Sept 25, 2008
posts: 23
votes: 0


Following on from a post about spider friendly JavaScript for collapsible content, I wondered if anybody had intentionally used uncrawlable pieces of JA or Ajax to prevent search engines crawling specific links.

As an example, many ecommerce solutions have multiple product links which screws up the balance of the internal linking structure. If you could restrict access to duplicate links and allow bots to crewel only a single text link (whilst not impacting usability) then the internal linking structure would be a lot healthier.

For the greater good make certain areas uncrawelable. Thoughts?
6:16 pm on Mar 16, 2010 (gmt 0)

Administrator from US 

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:June 21, 2004
posts:3080
votes: 67


#1 It depends on the situation.

#2 For some situations it can definitely help to block the spiders from reaching certain links & pages. I have done it on a few projects and have been rewarded with better rankings & traffic.
9:32 pm on Mar 16, 2010 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:May 14, 2008
posts: 3092
votes: 2


All my web sites (except where a customer requests otherwise) block or remove links from certain pages (eg contact forms) if a bot of any kind is detected, in addition to the page being blocked in robots.txt. If the bot hits the unlinked page then it gets a 405 returned.
9:09 am on Mar 17, 2010 (gmt 0)

New User

5+ Year Member

joined:Sept 25, 2008
posts:23
votes: 0


Thanks dstiles. I am thinking about preventing access through certain channels rather than preventing access to certain pages altogether where of course robots protocol could be used.

goodroi, what methods have you tested when preventing access through certain links and which have you seen the best results with?
7:47 am on Mar 18, 2010 (gmt 0)

New User

5+ Year Member

joined:Jan 21, 2009
posts:6
votes: 0


You can request Google to crawl specific sites/links for it's search engine by adding the URL into Google:

[google.co.uk...]

Entering your site here will get Google to crawl all the links it can find on that site.
12:15 pm on Mar 18, 2010 (gmt 0)

Administrator from US 

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:June 21, 2004
posts:3080
votes: 67


every situation is different. i have personally used robots.txt, metarobots and a few others. the method i use depends on the scale of the project and the preexisting structure.