homepage Welcome to WebmasterWorld Guest from 54.226.43.155
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Advertising / Paid Inclusion Engines and Topics
Forum Library, Charter, Moderator: open

Paid Inclusion Engines and Topics Forum

  posting off  
Meta Robots
to include or not?
CaveToad




msg:27822
 4:20 pm on Jan 17, 2001 (gmt 0)

OK,
I've been seeing some discussion of problems encountered when including the Meta Robots
tag in pages when submitted to AV.
Now is this true for
Meta Name=robots content=index,follow ( or noindex,nofollow ) or is there only a problem
with content=revist

I only use the index,follow and haven't seen any drops from AV but I haven't seen any overly deep spidering or indexing either. If it's a problem I could remove it. Any thoughts on the usefulness of this tag with AV lately? Help or Hindrance?

 

DaveAtIFG




msg:27823
 7:01 pm on Jan 17, 2001 (gmt 0)

Hey CaveToad, welcome to WmW! I use the index,follow version of the meta across the board with no problems. Although AV hasn't been real good to my sites lately, I'm convinced they ignore this tag. AV spiders use robots.txt and this tag is generally used by those spiders that don't.

CaveToad




msg:27824
 4:38 pm on Jan 18, 2001 (gmt 0)

Thanks that's good to know. Ok. Now time to go out an build a robots.txt, lucky me.

luckynh




msg:27825
 5:16 pm on Jan 18, 2001 (gmt 0)

>overly deep spidering or indexing

I have not see AV do ANY deep spidering all my log files show AV getting the index.htm & robots.txt then it's off an running not grabing any other page on the site?

Also, (off this topic) I have not see AV pick up anny new pages that were in 2001. (i.e. created a page in 12-3-00 and submited it, still not there... ) I also looked at other KW terms that might show new listings (i.e. football) and the newest date I've see in like 1-Dec-2001..
Any thoughts?...

CaveToad




msg:27826
 9:19 pm on Jan 18, 2001 (gmt 0)

I've seen pages getting picked up this year. Actually it's been quite good in AV so far. They just don't get picked up with any speed is the whole problem so i thought that maybe the robots meta was slowing things down or keeping the spider from indexing at all. That was my concern.

seth_wilde




msg:27827
 9:32 pm on Jan 18, 2001 (gmt 0)

Does anyone have any proven success with the meta robot tag? I was always under the belief that no engines used it.....

DaveAtIFG




msg:27828
 5:27 pm on Jan 19, 2001 (gmt 0)

INK seems to use it for one, Seth.

seth_wilde




msg:27829
 11:08 pm on Jan 19, 2001 (gmt 0)

Dave,

Does it actually speed up the spidering or make it crawl deeper or is it more just for disallowing the spider?

DaveAtIFG




msg:27830
 12:17 am on Jan 20, 2001 (gmt 0)

<rant>Nothing speeds up spidering!!</rant>
The noindex meta works with INK as a disallow, just like a "Disallow:" in robots.txt works with most spiders.

djuice999




msg:27831
 4:17 am on Jan 22, 2001 (gmt 0)

I will tell you from my experience that AV disregards all robots tags. In the past they have said they use the meta noindex,nofollow, but these days it may not be good to use them. I have several listings in AV for pages on my site that have had the noindex,nofollow on them from the begining. I've heard many statements from other Webmasters that using any meta robots tag is not good for AV and many others. Judge for yourself, but I no longer use them.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Advertising / Paid Inclusion Engines and Topics
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved