Forum Moderators: open
From W3C Workshop:
“This tag is meant to provide users who cannot control
the robots.txt file at their sites. It provides a last chance to keep their content out of search services. It was decided not to add syntax to allow robot specific permissions within the meta-tag.”
When I looked through the pages in my site, some content editors have implemented it wrong by closing quotes after INDEX. <META NAME="Robots" CONTENT="INDEX", "FOLLOW"> This may cause SE's to miss the "FOLLOW".
Since I have control of robots.txt, should I eliminate <META NAME="Robots" CONTENT="INDEX, FOLLOW"> or fix it?
The default behavior of a spider is to index and follow links. Both robots.txt and the robots meta tag are used when you want to modify the default behavior, i.e. to prevent indexing or spidering.
If you disallow Google from spidering a page using robots.txt, Google will still list a link to that page (without spidering it) if it finds a link to that page anywhere. You can find such links by clicking on the "[ More results from www.domain.com ]" link in the Google results. They are shown as a URL only, with no title or description - as expected since Googlebot did not spider the page.
While this is usually not a problem, in some cases it can be. The only way I have found to tell Google "Please don't mention this page at all" is to allow the page to be spidered in robots.txt, and then include the <meta name="robots" content="noindex,nofollow"> html tag.
Another consideration is that one of the search engines (Ink?) claims, or used to claim, that their default behaviour was "index,nofollow", thus requiring you to put "index,follow" in your robots tag if your wanted a deep-crawl.
To address your root problem, do a search for freeware/shareware multi-file editors and utilities. I have used a freeware program called "Multi search and replace" under Windoze to make site-wide changes on small sites. It is crude, and nowhere near as powerful as a Unix solution, but it works.
HTH,
Jim
