Robots.txt Vs META name="robots"

Which is better?

[Fuzzi_Bear]

Up til now, i've been using the following on all my sites: <META name="robots" content="index,follow">

On the pages that i want indexed of course. Otherwise i write "noindex"

But is it best to use a Robots.txt file instead? is there any major difference?

thanks.

[encyclo]

The advantage with using a robots.txt is that you can exclude whole directories or groups of files and the spiders will not fetch any of the pages within. If you are using a meta noindex then the spider has to fetch the page (using your bandwidth) to see whether it can index it or not. The meta noindex is useful of you have a specifiic set of files where it would be too difficult to list them in a robots.txt.

One note: if you want to allow indexing you do not need to specify a meta robots tag at all index and follow is the default. Just add the meta robots when you want to exclude pages, not to include them.

[Fuzzi_Bear]

Thanks. got that.

[Dijkgraaf]

Also robots.txt will stop well behave bots requesting the resource in the first place, whereas with the Meta tags the bot has to request the page to read them.

Some search engines will have a URL only listing for URL's that they found but are disallowed in robots.txt, however if you only have the META tags disallowing indexing of the page they generally don't list the URL. However as the URL only listings only come up if you do a search for all URL's of a site, this isn't such a big issue for most people.