Meta Robots

to include or not?

[CaveToad]

OK,
I've been seeing some discussion of problems encountered when including the Meta Robots
tag in pages when submitted to AV.
Now is this true for
Meta Name=robots content=index,follow ( or noindex,nofollow ) or is there only a problem
with content=revist

I only use the index,follow and haven't seen any drops from AV but I haven't seen any overly deep spidering or indexing either. If it's a problem I could remove it. Any thoughts on the usefulness of this tag with AV lately? Help or Hindrance?

[DaveAtIFG]

Hey CaveToad, welcome to WmW! I use the index,follow version of the meta across the board with no problems. Although AV hasn't been real good to my sites lately, I'm convinced they ignore this tag. AV spiders use robots.txt and this tag is generally used by those spiders that don't.

[CaveToad]

Thanks that's good to know. Ok. Now time to go out an build a robots.txt, lucky me.

[luckynh]

>overly deep spidering or indexing

I have not see AV do ANY deep spidering all my log files show AV getting the index.htm & robots.txt then it's off an running not grabing any other page on the site?

Also, (off this topic) I have not see AV pick up anny new pages that were in 2001. (i.e. created a page in 12-3-00 and submited it, still not there... ) I also looked at other KW terms that might show new listings (i.e. football) and the newest date I've see in like 1-Dec-2001..
Any thoughts?...

[CaveToad]

I've seen pages getting picked up this year. Actually it's been quite good in AV so far. They just don't get picked up with any speed is the whole problem so i thought that maybe the robots meta was slowing things down or keeping the spider from indexing at all. That was my concern.

[seth_wilde]

Does anyone have any proven success with the meta robot tag? I was always under the belief that no engines used it.....

[DaveAtIFG]

INK seems to use it for one, Seth.

[seth_wilde]

Dave,

Does it actually speed up the spidering or make it crawl deeper or is it more just for disallowing the spider?

[DaveAtIFG]

<rant>Nothing speeds up spidering!!</rant>
The noindex meta works with INK as a disallow, just like a "Disallow:" in robots.txt works with most spiders.

[djuice999]

I will tell you from my experience that AV disregards all robots tags. In the past they have said they use the meta noindex,nofollow, but these days it may not be good to use them. I have several listings in AV for pages on my site that have had the noindex,nofollow on them from the begining. I've heard many statements from other Webmasters that using any meta robots tag is not good for AV and many others. Judge for yourself, but I no longer use them.