Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

Meta Robots

to include or not?



4:20 pm on Jan 17, 2001 (gmt 0)

I've been seeing some discussion of problems encountered when including the Meta Robots
tag in pages when submitted to AV.
Now is this true for
Meta Name=robots content=index,follow ( or noindex,nofollow ) or is there only a problem
with content=revist

I only use the index,follow and haven't seen any drops from AV but I haven't seen any overly deep spidering or indexing either. If it's a problem I could remove it. Any thoughts on the usefulness of this tag with AV lately? Help or Hindrance?


7:01 pm on Jan 17, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Hey CaveToad, welcome to WmW! I use the index,follow version of the meta across the board with no problems. Although AV hasn't been real good to my sites lately, I'm convinced they ignore this tag. AV spiders use robots.txt and this tag is generally used by those spiders that don't.


4:38 pm on Jan 18, 2001 (gmt 0)

Thanks that's good to know. Ok. Now time to go out an build a robots.txt, lucky me.


5:16 pm on Jan 18, 2001 (gmt 0)

10+ Year Member

>overly deep spidering or indexing

I have not see AV do ANY deep spidering all my log files show AV getting the index.htm & robots.txt then it's off an running not grabing any other page on the site?

Also, (off this topic) I have not see AV pick up anny new pages that were in 2001. (i.e. created a page in 12-3-00 and submited it, still not there... ) I also looked at other KW terms that might show new listings (i.e. football) and the newest date I've see in like 1-Dec-2001..
Any thoughts?...


9:19 pm on Jan 18, 2001 (gmt 0)

I've seen pages getting picked up this year. Actually it's been quite good in AV so far. They just don't get picked up with any speed is the whole problem so i thought that maybe the robots meta was slowing things down or keeping the spider from indexing at all. That was my concern.


9:32 pm on Jan 18, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Does anyone have any proven success with the meta robot tag? I was always under the belief that no engines used it.....


5:27 pm on Jan 19, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

INK seems to use it for one, Seth.


11:08 pm on Jan 19, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member


Does it actually speed up the spidering or make it crawl deeper or is it more just for disallowing the spider?


12:17 am on Jan 20, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

<rant>Nothing speeds up spidering!!</rant>
The noindex meta works with INK as a disallow, just like a "Disallow:" in robots.txt works with most spiders.


4:17 am on Jan 22, 2001 (gmt 0)

I will tell you from my experience that AV disregards all robots tags. In the past they have said they use the meta noindex,nofollow, but these days it may not be good to use them. I have several listings in AV for pages on my site that have had the noindex,nofollow on them from the begining. I've heard many statements from other Webmasters that using any meta robots tag is not good for AV and many others. Judge for yourself, but I no longer use them.

Featured Threads

Hot Threads This Week

Hot Threads This Month