homepage Welcome to WebmasterWorld Guest from 54.197.94.241
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
The best way to add uncrawlable links?
Following on from a post about spider friendly JavaScript for collapsible c
jonny0000




msg:4098836
 3:34 pm on Mar 16, 2010 (gmt 0)

Following on from a post about spider friendly JavaScript for collapsible content, I wondered if anybody had intentionally used uncrawlable pieces of JA or Ajax to prevent search engines crawling specific links.

As an example, many ecommerce solutions have multiple product links which screws up the balance of the internal linking structure. If you could restrict access to duplicate links and allow bots to crewel only a single text link (whilst not impacting usability) then the internal linking structure would be a lot healthier.

For the greater good make certain areas uncrawelable. Thoughts?

 

goodroi




msg:4098954
 6:16 pm on Mar 16, 2010 (gmt 0)

#1 It depends on the situation.

#2 For some situations it can definitely help to block the spiders from reaching certain links & pages. I have done it on a few projects and have been rewarded with better rankings & traffic.

dstiles




msg:4099086
 9:32 pm on Mar 16, 2010 (gmt 0)

All my web sites (except where a customer requests otherwise) block or remove links from certain pages (eg contact forms) if a bot of any kind is detected, in addition to the page being blocked in robots.txt. If the bot hits the unlinked page then it gets a 405 returned.

jonny0000




msg:4099393
 9:09 am on Mar 17, 2010 (gmt 0)

Thanks dstiles. I am thinking about preventing access through certain channels rather than preventing access to certain pages altogether where of course robots protocol could be used.

goodroi, what methods have you tested when preventing access through certain links and which have you seen the best results with?

jameswsparker




msg:4100173
 7:47 am on Mar 18, 2010 (gmt 0)

You can request Google to crawl specific sites/links for it's search engine by adding the URL into Google:

[google.co.uk...]

Entering your site here will get Google to crawl all the links it can find on that site.

goodroi




msg:4100276
 12:15 pm on Mar 18, 2010 (gmt 0)

every situation is different. i have personally used robots.txt, metarobots and a few others. the method i use depends on the scale of the project and the preexisting structure.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved