Welcome to WebmasterWorld Guest from 34.204.203.142

Forum Moderators: goodroi

Message Too Old, No Replies

Robots.txt and iframes

How to disallow iframes using robots.txt?

     
9:11 am on Feb 10, 2008 (gmt 0)

Preferred Member from CH 

10+ Year Member

joined:Mar 10, 2004
posts:429
votes: 0


Hi,

I'm running a few iframes on specific pages on a site and Id like to prevent googlebot, slurp etc from crawling the link embedded in the iframe.

How do I do this?

Thanks in advance.

5:25 am on Feb 11, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member jomaxx is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Nov 6, 2002
posts:4768
votes: 0


Add the URLs of the iframe pages to the robots.txt list of exclusions. Or stick all the iframe pages in a separate directory & block the whole directory.
10:25 am on Feb 11, 2008 (gmt 0)

Preferred Member from CH 

10+ Year Member

joined:Mar 10, 2004
posts:429
votes: 0


Thanks Jomaxx.

Would this work for external links in the iframes?:

Disallow: / [outwardlinkiniframe1.html...]
Disallow: / [outwardlinkiniframe2.html...]

3:16 pm on Feb 11, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member jimbeetle is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Oct 26, 2002
posts:3295
votes: 9


No, you can't use robots.txt on one site to block spiders from crawling pages on a different site. If you control the external site you can use robots.txt on that site to block spiders.
8:02 am on Feb 13, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member jomaxx is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Nov 6, 2002
posts:4768
votes: 0


There are two cases here...

1. You have a standard HREF link that opens in the iframe. In this case, I think this is a situation for rel=nofollow.

2. The URL is the default value specified in the iframes tag itself. Given that Google will even try to spider URLs found in Javascript, they would probably try to spider this as well. But IMO it's likely there would be no PR or anchor link value associated with this. If you want to be extra sure, set the default URL to "" and use Javascript to load that URL in the frame. Be sure to munge the URL so that the spider doesn't recognize it.

11:00 pm on Feb 14, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member jimbeetle is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Oct 26, 2002
posts:3295
votes: 9


Hmmm, yeah, interesting ideas, jomaxx. But if foxtunes doesn't control the external site -- which it does sound like -- then there's really no control of whether those iframed pages are going to be crawled or not.

Not sure exactly what you intend to accomplish here, foxtunes. Is it to hide the link or actually prevent the iframe bits from being crawled?