Page is a not externally linkable
- Marketing and Biz Dev
-- Cloaking
---- Cloaking: Characteristics By Search Engine

Air - 6:35 am on Mar 5, 2001 (gmt 0)


Although not a definitive list, here is a breakdown of search engine characteristics useful for having cloaked pages live peacefully in the respective indexes of the following search engines.

Your additions, corrections, and suggestions are welcome.

Search Engine: Alta Vista
Stock UA: Scooter
URL: [altavista.com]

Sensitivity To Spam:
Very sensitive to spam. Pages will be removed or entire domain will be banned.

Indexing Criteria:

  • Content served to Alta Vista spiders must be very relevant to the content shown regular visitors.
  • No penalty for cloaking is applied if this criteria is met.

    Spidering Frequency:
    Will spider pages submitted through Add URL page. Major crawls occur approx. twice a year.

    IP Address Stability:
    Infrequent introduction of new IP Addresses for it's spiders.

    Decloaking Hazards:

  • "Translate" link carries AltaVista IP Address
  • Will spider using non stock User Agent.
  • Will use IP Addresses from EXODUS for spidering at times.
  • Tends to introduce new IP Addresses at major crawl times.

    Search Engine: Excite
    Stock UA: ArchitextSpider
    URL: [excite.com]

    Sensitivity To Spam:
    Somewhat sensitive to spam. Slow to remove pages in violation. Will ban domains in some spam cases.

    Indexing Criteria:

  • Content served to Excite spiders should be relevant to the content shown regular visitors.
  • No penalty for cloaking is applied if this criteria is met.
  • Appears to favor sites with a non shared IP.

    Spidering Frequency:
    Sporadic spidering.

    IP Address Stability:
    Infrequent introduction of new IP Addresses for it's spiders.

    Decloaking Hazards:

  • May spider out of @home IP range (not confirmed)
  • Will use IP Addresses from EXODUS for spidering at times.

    Search Engine: Northern Light
    Stock UA: Gulliver/x.x
    URL: [northernlight.com]

    Sensitivity To Spam:
    Not sensitive.

    Indexing Criteria:

  • Content served to Northern Light spiders should be relevant to the content shown regular visitors. Northern Light will tolerate (read may be necessary) a much higher degree of keyword repetition and overall keyword density than other spidering engines.
  • No penalty for cloaking is applied if this criteria is met.

    Spidering Frequency:
    Infrequent spidering.

    IP Address Stability:
    Rarely introduces new IP Addresses for it's spiders.

    Decloaking Hazards:

  • Will use IP Addresses from EXODUS for spidering at times.

    Search Engine: Inktomi
    Stock UA: Slurp
    URL: [hotbot.com] [canada.com] [anzwers.com] (<--Partial list)

    Sensitivity To Spam:
    Sensitive. Will bury pages that show different content per visit, since it's spiders visit often, the same content should be shown it's spiders on short successive requests by it's multitude of spiders.

    Indexing Criteria:

  • Content served to Inktomi spiders should be relevant to the content shown regular visitors. Inktomi will tolerate some degree of higher keyword repetition and overall keyword density, but not irrelevant content. Will accept cloaked pages through paid placement as well as free submission.
  • Spider is occasionally tripped up by sites with shared IP's or bad DNS entries.
  • No penalty for cloaking is applied if these criteria are met.
  • Currently burying pages that are submitted through the free ADD URL page.

    Spidering Frequency:
    Annoyingly frequent. Will generally only spider submitted pages, rarely following links. Multiple spiders will usually arrive requesting the same page, with each spider using a different IP Address and a mix of User Agents, the index page tends to be requested most.

    IP Address Stability:
    Frequent introduction of new IP Addresses for it's spiders, although lately the frequency is declining.

    Decloaking Hazards:

  • Multiple spider visits requesting the same page within short time periods.
  • Will use IP Addresses from EXODUS for spidering at times.
  • Huge number of IP's are assigned to it's spiders, making IP Address list maintenance more difficult.
  • Uses a variety of User Agents including the standard browser agent Mozilla/3.0

    Search Engine: Lycos
    Stock UA: Lycos_Spider_(T-Rex)
    URL: [lycos.com]

    Sensitivity To Spam:
    Moderately sensitive.

    Indexing Criteria:

  • Content served to Lycos spiders should be relevant to the content shown regular visitors.
  • No penalty for cloaking is applied if this criteria is met.

    Spidering Frequency:
    Sporadic spidering.

    IP Address Stability:
    Infrequent introduction of new IP Addresses for it's spiders.

    Decloaking Hazards:

  • Will use IP Addresses from EXODUS for spidering at times.

    Search Engine: FAST/Alltheweb
    Stock UA: FAST-WebCrawler/x.x
    URL: [alltheweb.com]

    Sensitivity To Spam:
    Moderately sensitive, but becoming more sensitive to it.

    Indexing Criteria:

  • Content served to Fast/Alltheweb spiders should be relevant to the content shown regular visitors.
  • No penalty for cloaking is applied if this criteria is met.

    Spidering Frequency:
    Infrequent but heavy spidering, this spider is becoming better behaved in this regard.

    IP Address Stability:
    Infrequent introduction of new IP Addresses for it's spiders.

    Decloaking Hazards:

  • Will use IP Addresses from EXODUS for spidering at times.

    Search Engine: Google
    Stock UA: Googlebot/x.x
    URL: [google.com]

    Sensitivity To Spam:
    Sensitive(Moderately and climbing), becoming more sensitive to it.

    Indexing Criteria:

  • Content served to Google spiders should be relevant to the content shown regular visitors.
  • Cloaking on non promotional domains tends to do better than straight promotional domains. Likely due to the boost from a real domain appearing in directory listings.
  • All cloaked pages should have links to each other as a "real" site would.
  • No penalty for cloaking is applied if this criteria is met.

    Spidering Frequency:
    Adequate spidering, and fairly thorough. Initial visit is usually light, with a through follow up spidering after that. Tends to repeat this pattern monthly. Very thorough spider.

    IP Address Stability:
    Frequent introduction of new IP Addresses for it's spiders.

    Decloaking Hazards:

  • Will use IP Addresses from EXODUS for spidering at times.
  • Will use non stock user agent.
  • Caches the spidered page by default unless prevented by a "NOCACHE" tag.
  • Not a decloaking hazard but be forewarned, appears responsive to user complaints for pages that violate it's spam guidelines.

    Directories: Yahoo! - LookSmart -- Open Directory Project
    Stock UA: Not Relevant
    URL: [yahoo.com] -- [looksmart.com] -- [dmoz.org]

    Sensitivity To Spam:
    Very sensitive - sites are human reviewed.

    Indexing Criteria:

  • These are not spidering engines, therefore no indexing takes place. They should review the pages you show regular visitors, follow their rules exactly when submitting a site for review.
  • No penalty for cloaking, due to the nature of review, directories are largely oblivious to cloaking.

    Spidering Frequency:
    Spiders from directories (that use spiders) perform link checking duties only.

    IP Address Stability:
    Not Relevant.

    Decloaking Hazards:
    Not Relevant.

    Thread source:: http://www.webmasterworld.com/cloaking/144.htm
    Brought to you by WebmasterWorld: http://www.webmasterworld.com