Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

Cloaking Gone Mainstream

Languages, Agents, Doc format, - cloaking is everywhere.



10:26 pm on Jan 30, 2003 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

Cloaking has taken on so many new meanings and styles over the last few years that we are left scratching our heads as to what cloaking really means. Getting two people to agree on a definition is nearly impossible with all the agent, language, geo targeting, and device specific page generation going on today. It is so prevalent, that it is difficult to find a site in the Alexa top 500 that isn't cloaking in one form or another.

This all came up for us in mid December when right at the height of the Christmas ecommerce season, a friends European site was banned or penalized by a search engine. After numerous inquiries, it was learned that the surprising reason for it was cloaking. I got asked to take a look at the site and figure out where their was a problem. The site owner didn't even know what cloaking was, let alone practice it.

I determined that his off-the-shelf server language and browser content delivery program was classifying search engines as a text browser and delivering them a text version of the page. In it's default configuration, this 5 figure enterprise level package classified anything that wasn't IE, Opera, or Netscape as a text browser and generated a printer friendly version of the page that was pure text.

We explained to the SE just what the situation was and they agreed agreed and took off the penalty after we said we'd figure out a way around the agent part. Unfortunately, the package had all but compiled in the agent support and they were surprised when we informed them about it. What was even better was looking around some fortune 500 companies that run the same software to find three entire sites that were in effect "cloaked" - they didn't have a clue.

In the end we solved the problem with another piece of software that would exchange the agent that the site delivery program was seeing. Yep, we installed cloaking software.

So lets have a little run down of the current state of cloaking in it's various forms:

We've talked a bit about about agent based cloaking recently [webmasterworld.com].

Search Engines Endorse Web Services Cloaking:

Cloaking has become just varying shades of gray. We now have instances where search engines themselves endorse cloaking (xml feeds) and in some instances are giving out cloaking software to deliver those xml feeds.

That has resulted in pages intended (cloaked) for one search engine being indexed by another search engine. There have been occasions where this endorsed content has been banned or penalized by another search engine.

Geographic IP Delivery:

Language translations have been a hot topic for the last year. Most major sites now geographic deliver content in one form or another. Hardly a month goes by when someone doesn't scream, I can't get to Google.com because they are transparently redirected to a local tld. You will also find those same search engines custom tailoring results for that IP address (eg: personalized content generation). You can see the effect your self by changing your language preferences on a few search engines that offer the feature.

One Browser Web:

The recent history of major browsers is summed up in IE4-6, and Netscape 3-7. There is also a large 2nd tier of browsers: Opera, Lynx, Icab, and Mozilla.

All of these agents support different levels of code and standards. They also have inherent bugs related to page display. If you are a web designer, you could get a degree in the various browser differences of CSS and HTML alone.

Just when we are starting to think in terms a one browser web, along comes a whole new set of browsers to consider: Set Top Boxes, Cell Phones, PDA's, and other Mobile Devices. These all have varying degrees of support for XML, XHTML, CSS2/3, and the web services protocol blizzard (eg: .net, soap...etal).

We've not even begun to talk about IE7 which is rumored to be in final internal beta testing. Then there is Apples new browser and the growing horde of Mozilla based clones. When you put it in those terms, our one browser web seems like a distant dream.

Delivering different content to these devices is a mission critical operation on many sites. Generating content for mobile devices is a vastly different proposition than delivering an xml feed to a search engine, or a css tricked out page for a leading edge browser.

Given that the combination of vistor ip and user agent can run into hundreds of possibilities, the only valid response is agent and ip cloaking.

Off the shelf cloaking goes mainstream.

There many off-the-shelf packages available today that include cloaking in one form or another. The perplexing part is that many sites are cloake in ways you wouldn't even know about. There are several major forum packages that cloak in some form or another.

I was at a forum this morning that was agent cloaking, and other that was language cloaking. In both cases, the webmasters don't even know that it is taking place - let alone have the tech knowledge to correct it.

Welcome to 2003 - Modern Era Of Search Engines.

This isn't the web of 98-99 where people would routinely get whisked away to some irrelevant site unrelated to their query. Todays search engines are vastly improved with most engine algorithms putting Q&A tests on every page they include. Those range from directory inclusion requirements, inbound link count and quality, to contextual sensitivity and even a pages reputation.

In this modern era where search engines now routinely talk about their latest off-the-page criteria algo advancements, it's clear that traditional se cloaking has little effect. It comes down to one simple fact, those that complain about SE cloaking are simply over looking how search engines work. The search engines have done a fantastic job at cleaning up their results programatically and by hand.

The most most fascinating thing about this new main stream cloaking is the situation where a site just classifies a search engine as a graphically challenged browser. In that case, cloaking becomes mostly a agent based proposition. The trouble starts when you throw in language delivery to the equation, or even delivering specific content as part of a search engine program.

All of these wide ranging factors combined to result in about 10 to the 4th power of page generation possibilities. In that situation, it almost becomes a necessity to put spiders into the all text browser category and deliver the same page to the se's that you deliver to cell phones or the Lynx browser.

Thus, we've come full circle on search engine cloaking. We no longer cloak to deliver custom content to search engines, we now cloak for the search engines to keep them from getting at our cloaked content for visitors.

<edit> cleaned up some typos and syntax errors</edit>

[edited by: Brett_Tabke at 6:15 am (utc) on Feb. 3, 2003]

Alan Perkins

2:35 pm on Feb 5, 2003 (gmt 0)

10+ Year Member

You say cloaking=spam. Most in this thread are trying to make a distinction between the two.

I was trying to keep spam out of this. But, since Google and Inktomi both categorically state that cloaking is not wanted, it seems they at least think it is spam. The question is, do they think of cloaking in the same terms as Brett's first post in this thread? I don't think they do. Inktomi, for example, even states:

If the purpose is to serve alternate pages to different human users, based on locality, browser, machine type etc., we do not consider that cloaking.

I think agreeing to Alans definition will hurt beginners more than clear anything up.

This isn't my definition. We are talking about what SEOs have traditionally meant when using the word "cloaking", and what search engines mean now when they use the word.

This forum was once called "Cloaking - Stealth", and was in "The SEO World" section of WebmasterWorld. When did that stop?

You guys are on a marketing campaign to change a words meaning.

No, not me. :) Look in the WebmasterWorld Cloaking Forum Charter [webmasterworld.com]:

Cloaking is delivering one version of a page to a search engine spider while serving users another version.

That's a little bit different to some of the things that have been described in this thread as cloaking.

Now look in the WebmasterWorld Cloaking Forum Library [webmasterworld.com] and you'll find a thread from Oct 2, 2000 called Comment on cloaking from a SE [webmasterworld.com]. I participate in that thread, and I'm using exactly the same definition of cloaking then as I'm using now, 2.5 years later. Note how everyone else in that thread is talking about IP delivery and UA delivery to search engines. That's what cloaking was and still is, as far as I am concerned. And I think, from their web sites, that Google and Inktomi at least agrees.


3:00 pm on Feb 5, 2003 (gmt 0)

10+ Year Member

That's what cloaking was and still is, as far as I am concerned. And I think, from their web sites, that Google and Inktomi at least agrees.

Actually, I spoke to Inktomi yesterday for the articles on this subject that I posted today. They agreed that XML feeds technically are cloaking. They also said that what is cloaking has changed so much because of the new technologies over the year. They are planning to go back and revise their guidelines to try and better explain what type of cloaking they don't like.

Alan Perkins

3:13 pm on Feb 5, 2003 (gmt 0)

10+ Year Member

They are planning to go back and revise their guidelines to try and better explain what type of cloaking they don't like.

LOL, OK then!

Strange thing is, I'm sure they only updated their Inktomi Content Guidelines [inktomi.com] recently, and on the same page they say both

What Inktomi Considers Unwanted
Some, but not all, examples of the more common types of pages that Inktomi does not want include:
Pages that give the search engine a different page than the public sees (cloaking)


Treatment of Paid Content
Inktomi designed its Index Connect and Search Submit programs to improve the quality of its search databases and thereby enhance the search user experience. Therefore, URLs submitted via Index Connect and Search Submit are subject to these rules and any other additional rules or policies adopted by Inktomi from time to time.

It will be interesting to see what they say now.

If things become clear and consistent for everybody, that will be just great! That was the intent of the article. IMO, though, things have always been clear and consistent up until now. :)

Alan Perkins

4:34 pm on Feb 5, 2003 (gmt 0)

10+ Year Member

Hi Danny

I just read your article, "Doorways Not Always Bad, At Inktomi". Very interesting!

As it relates to this thread, this line was relevant:

Inktomi now says Inktomi will be revising its guidelines about cloaking to add the concept of deception. Unfortunately, I couldn't get a good answer about what "deceptive" means.

IMO it means that the part of the algo that thinks it is evaluating the content that searchers will see is deceived, because it is actually evaluating content that searchers will never see.

With XML feeds the algo isn't deceived, because the algo knows what it is dealing with. Nothing is cloaked from the search engine. So XML feeds are not cloaking.

With cloaking as I have always accepted it to mean, the content that searchers see is cloaked from search engines. That part of the algo that evaluates what searchers will see is deceived.

However, if Inktomi wants to say that trusted feeds ARE cloaking, that's up to them! ;)


5:56 pm on Feb 5, 2003 (gmt 0)

10+ Year Member

IMO it means that the part of the algo that thinks it is evaluating the content that searchers will see is deceived, because it is actually evaluating content that searchers will never see.

Let me clarify. I don't know what they are going to consider deceptive because THEY don't know what they consider deceptive.

They seemed to think that if you sent a cloaked page that told the spider it was about Britney Spears but that the user saw a porn page, that would be deceptive.

They were entirely unclear about whether if you cloaked a page for their spider about Britney Spears, then deliverd a user to say a dynamically-generated page full of Flash content about Britney Spears, whether that would be deceptive.

In the example above, they said there are cases in BOTH paid and unpaid content where they have allowed cloaking to happen like this, because they decided there was no other way to get the content and that the user wasn't being deceived.

Overall, I think they are now pondering some of these issues anew for the first time. They said as much -- that cloaking has changed so much in terms of how it can be done and reasons why it is done that they need to figure out the right guidelines to provide.


7:46 am on Feb 6, 2003 (gmt 0)

10+ Year Member

I have to post this newsletter I received from webwhiz aka Jill, from her very own High Rankings site. Her glaring EXCLUSION of WebmasterWorld appears almost disrespectful and definitely evasive, as she most certainly wanted to spare her own members the drama she encountered here at WebmasterWorld.
Also makes me wonder in regards to her original intentions in posting to this thread. Exposure for Mr.Perkins or,,,something else?

This is an excerpt as it relates to this thread:

[edited by: engine at 12:51 pm (utc) on Feb. 6, 2003]
[edit reason] <snip> Please see TOS 9 & 10 [/edit]


9:54 am on Feb 6, 2003 (gmt 0)

I really dont see what the content of her latest newsletter has to do with any of this.

If people want to come to WebmasterWorld and hear an alternative view then they can do so.

What is the point of your post, other than to vilify and scandalise?

Anyhow, great thread all!

Id just like to say that debate is always a good thing, the status quo regarding cloaking is causing some people difficulties, and its good to see the issues aired and discussed.

I dont think it helps the webmaster community to have polarised conflicting definitions. Lets hope that the recent debates have helped bridge the gap.

Danny, Brett, Alan, Jill and many other well known speakers have contributed some fantastic insights and expositions that have, for me at least, helped clarify a few things.

For this, thanks. :)


12:44 pm on Feb 6, 2003 (gmt 0)

10+ Year Member

Fred, I posted a link from the forums thread to this forum, which is why I felt no need to post to this one in the newsletter also. I knew that people would click to here from there. In fact, I believe I posted to this thread a number of times in the other forum thread.

Perhaps you should have all the facts at hand before making assumptions.


7:43 am on Feb 7, 2003 (gmt 0)

10+ Year Member

I can't believe this topic has gone on for so many pages, when it's simple semantics. Some users refer to all cloaking as cloaking; some users referring to cloaking are actually referring to deceptive cloaking, i.e., SPAM.
I tend to use the term the same way Jill does. If you someone is talking about IP delivery, I feel they should say IP deliver. If they are targeting just the Googlebot, it's SPAM, because I won't see what Google indexed if I do click through to the page. I've seen a lot of cloaking aimed at Google, and so far all of the Google-targeted IP delivery I've seen is SPAM.
<Disclaimer> Just my .02 <Disclaimer>
This 129 message thread spans 5 pages: 129

Featured Threads

Hot Threads This Week

Hot Threads This Month