Robots.txt and iframes

How to disallow iframes using robots.txt?

[foxtunes]

Hi,

I'm running a few iframes on specific pages on a site and Id like to prevent googlebot, slurp etc from crawling the link embedded in the iframe.

How do I do this?

Thanks in advance.

[jomaxx]

Add the URLs of the iframe pages to the robots.txt list of exclusions. Or stick all the iframe pages in a separate directory & block the whole directory.

[foxtunes]

Thanks Jomaxx.

Would this work for external links in the iframes?:

Disallow: / [outwardlinkiniframe1.html...]
Disallow: / [outwardlinkiniframe2.html...]

[jimbeetle]

No, you can't use robots.txt on one site to block spiders from crawling pages on a different site. If you control the external site you can use robots.txt on that site to block spiders.

[jomaxx]

There are two cases here...

1. You have a standard HREF link that opens in the iframe. In this case, I think this is a situation for rel=nofollow.

2. The URL is the default value specified in the iframes tag itself. Given that Google will even try to spider URLs found in Javascript, they would probably try to spider this as well. But IMO it's likely there would be no PR or anchor link value associated with this. If you want to be extra sure, set the default URL to "" and use Javascript to load that URL in the frame. Be sure to munge the URL so that the spider doesn't recognize it.

[jimbeetle]

Hmmm, yeah, interesting ideas, jomaxx. But if foxtunes doesn't control the external site -- which it does sound like -- then there's really no control of whether those iframed pages are going to be crawled or not.

Not sure exactly what you intend to accomplish here, foxtunes. Is it to hide the link or actually prevent the iframe bits from being crawled?