| 7:01 pm on Jan 17, 2001 (gmt 0)|
Hey CaveToad, welcome to WmW! I use the index,follow version of the meta across the board with no problems. Although AV hasn't been real good to my sites lately, I'm convinced they ignore this tag. AV spiders use robots.txt and this tag is generally used by those spiders that don't.
| 4:38 pm on Jan 18, 2001 (gmt 0)|
Thanks that's good to know. Ok. Now time to go out an build a robots.txt, lucky me.
| 5:16 pm on Jan 18, 2001 (gmt 0)|
>overly deep spidering or indexing
I have not see AV do ANY deep spidering all my log files show AV getting the index.htm & robots.txt then it's off an running not grabing any other page on the site?
Also, (off this topic) I have not see AV pick up anny new pages that were in 2001. (i.e. created a page in 12-3-00 and submited it, still not there... ) I also looked at other KW terms that might show new listings (i.e. football) and the newest date I've see in like 1-Dec-2001..
| 9:19 pm on Jan 18, 2001 (gmt 0)|
I've seen pages getting picked up this year. Actually it's been quite good in AV so far. They just don't get picked up with any speed is the whole problem so i thought that maybe the robots meta was slowing things down or keeping the spider from indexing at all. That was my concern.
| 9:32 pm on Jan 18, 2001 (gmt 0)|
Does anyone have any proven success with the meta robot tag? I was always under the belief that no engines used it.....
| 5:27 pm on Jan 19, 2001 (gmt 0)|
INK seems to use it for one, Seth.
| 11:08 pm on Jan 19, 2001 (gmt 0)|
Does it actually speed up the spidering or make it crawl deeper or is it more just for disallowing the spider?
| 12:17 am on Jan 20, 2001 (gmt 0)|
<rant>Nothing speeds up spidering!!</rant>
The noindex meta works with INK as a disallow, just like a "Disallow:" in robots.txt works with most spiders.
| 4:17 am on Jan 22, 2001 (gmt 0)|
I will tell you from my experience that AV disregards all robots tags. In the past they have said they use the meta noindex,nofollow, but these days it may not be good to use them. I have several listings in AV for pages on my site that have had the noindex,nofollow on them from the begining. I've heard many statements from other Webmasters that using any meta robots tag is not good for AV and many others. Judge for yourself, but I no longer use them.