Welcome to WebmasterWorld Guest from 23.23.46.20

Forum Moderators: goodroi

Blocking all URLS to a website using the robots.txt file?

   
2:01 am on Jan 4, 2006 (gmt 0)

5+ Year Member



I run a real estate website and we link out to a MLS system on almost every page of the website. Is there a way to use the robots.txt file to block spiders from spidering these links? Each link is a different url but from the same domain.

Thanks

Jared

10:05 am on Jan 6, 2006 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member



No, not via robots.txt

However you can do it via either
1) A the robots meta tag
<meta name="robots" content="index,nofollow">
This tells a bot to index the page, but not any links it find in that page.
Advantage: you only have to do it once per page in the header.
Disadvantage: This will be for all links in the page, it is not selective to outbound links, but will include internal ones.
See [robotstxt.org...] for more details
2) The attribute nofollow
e.g <a href="http://www.example.com" rel="nofollow">
See [linktutorial.com...]
Advantage: It allows you to selectively tell a bot not to follow certain links
Disadvantage: You have to do it for each link.

Depending on what you have and are trying to achieve will determine which one you should use.

 

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month