homepage Welcome to WebmasterWorld Guest from 54.196.189.229
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Blocking all URLS to a website using the robots.txt file?
scubajared




msg:1528315
 2:01 am on Jan 4, 2006 (gmt 0)

I run a real estate website and we link out to a MLS system on almost every page of the website. Is there a way to use the robots.txt file to block spiders from spidering these links? Each link is a different url but from the same domain.

Thanks

Jared

 

Dijkgraaf




msg:1528316
 10:05 am on Jan 6, 2006 (gmt 0)

No, not via robots.txt

However you can do it via either
1) A the robots meta tag
<meta name="robots" content="index,nofollow">
This tells a bot to index the page, but not any links it find in that page.
Advantage: you only have to do it once per page in the header.
Disadvantage: This will be for all links in the page, it is not selective to outbound links, but will include internal ones.
See [robotstxt.org...] for more details
2) The attribute nofollow
e.g <a href="http://www.example.com" rel="nofollow">
See [linktutorial.com...]
Advantage: It allows you to selectively tell a bot not to follow certain links
Disadvantage: You have to do it for each link.

Depending on what you have and are trying to achieve will determine which one you should use.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved