Does robots.txt override meta tag?

[kira]

I uploaded robots.txt file to the domains I don't want spidered. These domains all have the meta tag <META NAME="ROBOTS" CONTENT="INDEX,FOLLOW">. I'm wondering if the robots.txt will override the meta tag, or whether I will need to change the tag to noindex,nofollow....

[TallTroll]

Depends on the content of your robots.txt. If you block access to a given page via robots.txt, but the page contains a meta robots tag, the meta is useless, because the spider will never request the page, and will therefore not see the tag.

[bird]

A well behaved spider will not even look at the pages that are excluded by robots.txt. In other words, for those, it doesn't matter at all what they could find there if they looked. However, there's the off chance that a not quite so well behaved spider might ignore robots.txt and only look at the meta tags, so you don't want to put contradicting information there.

In fact, any kind of policy information (robots.txt and the robots meta tag are nothing else) should always be consistent across all the channels you use. What point is there in saying "no" with your left mouth, and "yes" with the right?

The only situation where this could make any sense would be when excluding only one (or several) specific spiders from robots.txt and allowing all others. But the phrasing of your question makes me think that you either want to allow all of them or none at all.

[kira]

I am dealing with about 100 sites that already have the meta tag INDEX,FOLLOW.

I thought that by uploading the robots.txt

User-agent: *
Disallow: /

I wouldn't have to change all the meta tags on every site (?)

[ciml]

You should be fine, Kira. I'm not sure about the other way round, but as bird writes, Google won't even load your pages so the META robots can't apply.

[TallTroll]

That file will lock out any spider that obeys the robots.txt. They shouldn't look at anything

Of course, not all spiders are that well behaved...

[kira]

Thank you for taking the time to respond to my post! Seems I'm good to go, then!;-)

[stevenha]

What about this "reverse" situation:
Suppose the robots.txt file contains a single byte (spacebar), indicating that all pages are OK to spider,
Will the SE conclude the robots.txt should override a meta robots noindex,nofollow which might appear on an individual page? Or ( as I hope ), will the spider still respect the "noindex" tag on an individual page