Welcome to WebmasterWorld Guest from 54.167.102.69

Forum Moderators: goodroi

Message Too Old, No Replies

Does robots.txt override meta tag?

     
2:12 pm on May 2, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:Apr 25, 2002
posts:98
votes: 0


I uploaded robots.txt file to the domains I don't want spidered. These domains all have the meta tag <META NAME="ROBOTS" CONTENT="INDEX,FOLLOW">. I'm wondering if the robots.txt will override the meta tag, or whether I will need to change the tag to noindex,nofollow....
2:31 pm on May 2, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 6, 2001
posts:880
votes: 0


Depends on the content of your robots.txt. If you block access to a given page via robots.txt, but the page contains a meta robots tag, the meta is useless, because the spider will never request the page, and will therefore not see the tag.
2:37 pm on May 2, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Aug 10, 2001
posts:1550
votes: 10


A well behaved spider will not even look at the pages that are excluded by robots.txt. In other words, for those, it doesn't matter at all what they could find there if they looked. However, there's the off chance that a not quite so well behaved spider might ignore robots.txt and only look at the meta tags, so you don't want to put contradicting information there.

In fact, any kind of policy information (robots.txt and the robots meta tag are nothing else) should always be consistent across all the channels you use. What point is there in saying "no" with your left mouth, and "yes" with the right?

The only situation where this could make any sense would be when excluding only one (or several) specific spiders from robots.txt and allowing all others. But the phrasing of your question makes me think that you either want to allow all of them or none at all.

2:58 pm on May 2, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:Apr 25, 2002
posts:98
votes: 0


I am dealing with about 100 sites that already have the meta tag INDEX,FOLLOW.

I thought that by uploading the robots.txt

User-agent: *
Disallow: /

I wouldn't have to change all the meta tags on every site (?)

3:25 pm on May 2, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member ciml is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 22, 2001
posts:3805
votes: 2


You should be fine, Kira. I'm not sure about the other way round, but as bird writes, Google won't even load your pages so the META robots can't apply.
3:26 pm on May 2, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 6, 2001
posts:880
votes: 0


That file will lock out any spider that obeys the robots.txt. They shouldn't look at anything

Of course, not all spiders are that well behaved...

3:36 pm on May 2, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:Apr 25, 2002
posts:98
votes: 0


Thank you for taking the time to respond to my post! Seems I'm good to go, then!;-)
7:12 pm on May 2, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:Apr 15, 2002
posts:172
votes: 0


What about this "reverse" situation:
Suppose the robots.txt file contains a single byte (spacebar), indicating that all pages are OK to spider,
Will the SE conclude the robots.txt should override a meta robots noindex,nofollow which might appear on an individual page? Or ( as I hope ), will the spider still respect the "noindex" tag on an individual page
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members