|Bye bye cloaking|
As of 12 noon local time, all cloaking on all our sites is fully removed. First time in over a year we've done that. Never been riskier. I just don't feel the pay off justifies the risk any longer. From Inktomi, Alta, Infoseek, and Excite, SE's spiders are no longer identifiable on a reliable basis.
Take it for what its worth, from the 'what I heard' file: Alta looks at sites for various red flags. When it see's a red flag, it throws it into a hooper and then spiders that site with a standard agent name from AOL based accounts - in other words, they pull the page out of the AOL cache and you have no traceable idea they were there.
Who is to say inktomi isn't doing the same thing with AOL since there are cache arrangments between the two and AOL is servicing Netfind to boot.
Also remember that it is email@example.com. You know, high speed net access. connect the dots.
Cloaking is dead.
So, the really interesting thing will be, what it's like when you are reindexed.
Most of our cloaking was taken off last feb and we noticed very little movement. The biggest was with Inktomi - most pages went up.
Well, I beg to differ on that one, Brett:
if I see what the domains we're working for (our own and clients'),
I'd say cloaking has never been more alive and kicking!
Then, I don't see any indication whatsoever that search engines
(including AV and Ink) are blackmarking cloaking per se.
They may be working overtime to eliminate the spam (however defined),
but if what you said were true we ought to have lost more domains
than you shake a cursor at by now.
As for spider detection: it's only reliable if you work it by IP
and this requires a terrible amount of manual labor, expertise and
sound judgment all the way.
Human counter checking via a standard set up can't be ruled out,
of course, unless you're privy to the IPs used, which in the case
of AOL is useless because of the countless numbers of dynamic IPs
being assigned (one for each open browser window, basically).
Still - cloaking is the designer's and layouter's only truly effective
weapon against the dictatorship of the search engines whose algos
haven't really developed beyond the "text only" state they started
from a decade ago.
You may not particularly like splash or Flash index pages (I know I
it's the SEs' job to cope with that, not the webmasters' to comply with
their dumb and outdated tech.
And speaking from a pragmatic point of view: cloaking works - and
it's constantly being improved. Also, if you're worried about being
detected, simply cloak some doorway domains instead, something which
any good cloaking service would advise you to do anyway. (Nothing new
there, really: this used to be common practice long ago.) That way,
you'll only risk losing those. I haven't seen any indication to date
that this will backfire to your core domain as well.
Finally, as with everything in life: it's as much a question of not
overdoing it as anything else.
Fantomaster, I'm glad you finished your post with the last line.
It seems to me that the SEs will have to do some more shaking of the tree before cloaking in the old sense dies entirely. Why? There are just so many pages out there and it will take some time for everything to filter through. By then, I'd guess those "experts" on cloaking will have leaped ahead.
Personally, I hate the idea of cloaking fooling a surfer to vistit an irrelevant site - such as switching the surfer to a porn site. I've been a "victim" of this trick and it ####s! On the other hand, when there is a relevant page/site, I see no reason not to use the techniques at your disposal (cloaking or otherwise). They are not "illegal" techniques, but are contrary to the SEs Terms and conditions we all agree to when we use the SEs. Therefore - don't get caught out otherwise you'll pay the penalty.
I do not use cloaking because the majority of KW I focus on are not ferociously competitive (up to 1-million returns). I'd hate to be in the "real estate" sector where it's probably compulsory to use cloaking.
As surfers become more sophisticated at searching (ha ha) they will learn better techniques. If you ask ten ordinary surfers how to search properly, you get nine out of ten with little or poor knowledge.
I'm doing my best to educate my clients how to search.
Just my 2-cents worth.
I must agree with Brett in part. An older site (year and a half) that has been cloaked most of it's life has improved it's position on INK engines recently without cloaking. A newer site (six months and designed from the ground up to rank well, themes, keywords, etc) got listed well and immediately on INK without cloaking. An INK press release from a few months ago that included a bunch of statistics, "Number of cloaked sites detected" was one. Can't find the darn thing now though... My impression is that different INK partners penalize differently for cloaking.
Currently AV doesn't seem to be trying to detect cloaking that I've noticed. They are definitely not penalizing for discreet cloaking, optimizing a RELEVENT page, in my experience.
NL seems to be a mixed bag. It appears as if older pages in their index can cloak with impunity. New submissions appear to be checked for cloaking and penalized if detected.
I feel that as the major SEs become more sophisticated the value of cloaking will be further eroded. Fanto, don't get defensive. If you can remain ahead of the SEs technically, you may become a specialized service that an average cloaker will buy instead of DIY because it has become so labor intensive to keep up with the technology.
My own limited experience strongly suggests that links popularity carries substantially more weight then any page optimization cloaking provides with many of the major engines. I fully expect this to spread to all of the major SEs soon.
Not being defensive on our own score: we know pretty well where
we succeed and where not. But the web being the fairly hysterical
medium it is, it doesn't take too much to instigate (if unintentionally)
a witch hunt on cloakers. (Look at what happened to the term "spam"
in recent years: nowadays, hardly anyone seems to remember what spam
used to be defined as. Instead, what it *has* led to is the merry hunt
of the bigotted clueless, self-declared "web cops" decrying more or
less anything as spam which doesn't seem to fit into their particular
reality tunnel. The mechanisms at work here are basically the same which
in meatspace will provoke lynch mobs, pogroms etc.)
Look at all the discussion forums where people are throwing their weight
about shouting how "decidedly anti-cloaking" they are, and you can
see it coming ...
There's basically only two approaches towards search engine positioning:
1. You can succumb to their tyranny and inferior technology - more often
than not to the detriment of layout and design and, in many cases, even
content. (Database driven sites featuring dynamic content are a typical
case in point.)
2. You can exploit all possibilities at your disposal to get your sites
well ranked without foul compromises in design and layout, content, etc.
Personally, I detest irrelevant cloaking as much as the other guy, and I
find this sort of tactics plain dumb to boot: searching for auto parts,
I'm certainly not going to buy *anything* from some idiot redirecting me
to his porn pages. I don't take kindly to people trying to fool me, and I
think most surfers don't, either, at least not after having gained a
modicum of net experience to draw upon.
But then, of course, it's not a question of morals or ethical standards
(which, after all, are for anybody to define at their whimsy) - I'm only
concerned with what works and what doesn't. Which, of course, raises
the question of the working conditions involved, etc.
engine: I agree in full with your statement, and I can actuall show you
sites where we're achieving lots of good results with no (or hardly any)
cloaking involved, even in highly competitive areas.
Neither would I place my belief in anyone claiming that cloaking is the
one and only immaculate key to search engine positioning bliss - that just
But you're right about the real estate biz - refuse cloaking there, and you
might just as well drop out of the industry altogether. The same holds true
for lots of other industries, notably the adult content world, gambling,
online finance, and more. Here, it's simply "cloak or croak" ...
And Dave, yes: without wishing to unduly add to the mystique which still
surrounds cloaking for many, many webmasters - present company excepted,
of course - I agree that what we term "industrial-strength" cloaking
is becoming an ever more specialized realm which doesn't lend itself
easily (or, rather, safely) to the prevalent DIY ideology.
As for link popularity: it's sure doing wonders currently, but it remains
to be seen if it will ever become more than a mere fad. That's because
the basic logic underlying it is pretty flawed at best - it will only
favor big players in the long run; it's easy to counter, etc.
It works on the dubious principle that sixty billion flies simply can't
be wrong, which is why you, too, should favor horse dung for your diet ...
It may only be a matter of time till it evaporates as did overhyped
It's been a month since you launched this thread, Brett, and 4 months since you started de-cloaking. Any updates?
I've kept just a touch of cloaking for a specific engine; but, I've given up on the big two (alta/ink).
I just don't trust Alta not to screw around. I've checked and checked the close competition on alta, and the top sites also appear not to be cloaking, or they have gotten very good at it. why? They've certainly done something over there to atleast shake out the amature cloakers - you rarely get those various styles of redirects to base pages anymore from top-of-the-results page sites. I think it is indicative that they have cross referenced data from somewhere. Either from their own spiders or cross referencing data with dec's Mercator.
Futher down in the results I still can find a few redirects (a minor indicator of cloaking activity).
Ink? Little doubt that they are running shawdow spiders. They are xreferencing that data from somewhere. I think they are pulling data out of proxy caches from all those cache servers they own, lease, or operate for others like AOL (it is ink's pcache software running on aol). Our cloaked sites went doa in March. Turned off the cloak, and finally back into some decent rankings again (decent defined as top 50).
I've went back to cloaking for Fast at the moment. Don't know if that will last, but with Lycos now using Fast, I wanted to play hard. I just feel there is more untapped potential in Lycos and I wanted to get ready.
There is going to be a day here sometime this summer when Lycos drips with traffic again. The word about Fast/Lycos is slowly trickling out. As these things usually go, sooner-or-later, one of the big news outlets is going to pick it up and *bam*, mega traffic at Lycos. Lycos also has all that stuff going on in Europe with Terra and Fast, I want to be ready for those days.
Ya, I've always tried to ride the waves. Like the old Infoseek. I used to run a news and current events site when Infoseek was indexing instantly. That gave me a good insight into how people surf in mass on the spur of the moment. That is why I was ready when alta did the switch last year, why I played Google hard when they were getting all the free press, and Excite when @home took over. I just want to be ready for Lycos when it finally 'pops'.
I have a couple of pages still cloaked for Ink of data I don't want exposed but need the link reference.
I don't think I'll ever go back to cloaking on the big two (or excite). Never really cared for it in the first place.
Not all cloaking programs being created equal, would you care to relate some specs on the type of software you're using? E.g. does it work by IP delivery, if so: does it include IP ranges or single unique IPs only;
does it operate outside of cgi-bin, etc.
It may make all the difference, you know ...
Re cloaking and AV:
Every major effort I've had with AV and cloaking has been booted out. Yet at the same time, I have several small (older) cloaked pages in their database. I believe
that there is a size (or aggression) factor at AV to send up flags. There is a seo I know who only does mom and pop type of websites that have little volume. He has
been IP delivering for 1 1/2 years now and has never had a problem.
I believe there is probably a safe(r) level of cloaking that could be used in av, but in this case smallness is necessary. Maybe three a day, and it shouldn't be anything
that will send up any flags (i.e. viagra, sex, real-estate and such).
I am going to be trying a new delivery approach other then IP, host_name, or user agent and see what kind of longevity I have. BTW it isn't going to be the
robots.txt method discussed at search engine forums.
I am still having very good luck cloaking IP style with them.
Software: I do full name resolution on the fly on all our sites. I only use ip's when there is no other choice. I've never gotten in trouble in that respect. I feel perl IP based cloaking is too slow. You can do the dns in a fraction of the time it takes a loop of perl to walk through the possibilities. It really helps if the server itself does the dns and has that resolved before the script ever see's it.
Whenever there was a doubt (dns failure or suspected ip that wouldn't resolve), I let it have the real page.
We rarely expose cgi-bin on anything but specific user stuff that isn't meant to get indexed anyway (like stuff on se world). Did you notice I converted all this software here to run ssi - whew that was a job, but maybe spiders will take a shine. I really hadn't thought about it, but I think I'll wall off some of the alta-ink-google engine posts. My paranoia index has been steadly falling since the january surprise - that is usually the time you should really start to worry.
If I add a cloaking forum, would it go down in flames due to umm *ahem* cloaking software authors taking sniper shots at each other? (I would be the only moderator of it). I don't think Ralph was fortunate enough to see the cloaking thread in sef last summer - that was a real winner.
|Software: I do full name resolution on the fly on all our sites. |
IMO this isn't quite safe enough.
|I feel perl IP based cloaking is too slow. You can do the dns in a fraction of the time it takes a loop of perl to walk through the possibilities. It really helps if the server itself does the dns and has that resolved before the script ever see's it. |
Agreed to the latter - but if you work from a really good database - like the one I'm not allowed to mention here :) - the effect is quite minimal, even in flat format. Ours has some 145K currently which, on a good state-of-the-art system, is real fast.
Moreover, you could work it from a MySQL database which would boost performance further.
|Whenever there was a doubt (dns failure or suspected ip that wouldn't resolve), I let it have the real page. |
That, of course, is sound policy.
|If I add a cloaking forum, would it go down in flames due to umm *ahem* cloaking software authors taking sniper shots at each other? (I would be the only moderator of it). |
Mutual trashing of programs is a darn dumb marketing strategy at best: ego aside, who is it supposed to convince? If you do add a cloaking forum, I'd be the first to applaud you - the stuff currently available on sef is 90% tripe, and if you can calm the flamers it could be a real winner for everybody involved (and concerned).
|I don't think Ralph was fortunate enough to see the cloaking thread in sef last summer |
I was - but I had stopped posting on that forum by then. We were not amused ...
I have to admit that I'm not much more than an amatuer cloaker and it hasn't really gotten me in trouble at all.
I more or less use a server side, ASP based IP delivery (specific, not ranges) system and it works in AV especially.
I just recently got INK to list pages that are basically ASP pages with a quick, less than subtle redirect.
I'd like to perfect it. Which is a different post I guess, but since I have some success now in places where people have trouble, isnt that a start?
|If I add a cloaking forum, would it go down in flames due to umm *ahem* cloaking software authors taking sniper shots at each other? (I would be the only moderator of it). I don't think Ralph was fortunate enough to see the cloaking thread in sef last summer - that was a real winner. |
Brett, not sure if that is sarcasm (I think it is) but from what I recall the moderator took all the pot shots, the authors were pretty well behaved. keep on rockin' bud - good job with the new forum.
BTW I lost my id (special characters in it) got a new one now - when registering said "Air" was taken, so any posts from "Air" aren't me - cool...
<sarcasm>Weeeelll this is delightful.</sarcasm>
I have tried everything to get one of my client's site some lift on Excite, with no luck. I find out about this thing called "cloaking" and think I have found the answer to all of my search engine prayers.
And two weeks later, I find this forum and you all are saying that cloaking is on its way out, and that if I do it, my page will be killed.
/me bonks her head repeatedly on her desk.
Having read the quotes from search engine reps declaring they permanately ban sites for cloaking, and my not liking the results of cloaking abuse out there, I decided to not risk 4+ years worth of work and client base.
Still, I agree with a lot of what the Fant Meister says. I've been submitting sites since 1995 and have grown fed up with the lack of progess with search technology. If I see another top 50 results of all cloned doorway pages with no content I'm going to scream!
Overall, my advice tends to lean on the conservative approach. Not because I'm a wimp, but because people want to be found. Many want to do it without tricks. To do that, they have to build "dumb" sites that robots can do something with. It's a safe, uninspired approach. It also destroys the fun for Flash sites and design with an artistic, dynamic or think-outside-the-box attitude.
The latest slap in the face is Alta Vista banning websites for being identical to each other. Affiliate programs that allow affilates to use a cloned version of the "mother ship" site are getting killed with this wacky rule. Even the "mother ship" is banned. Ecommerce storefronts based on templates are in jeapordy. Just exactly how much risk are templates going to be in the future?
It's nice to see some frank discussion on these matters. Anyone here a search engine CEO?
Brett and Fanto, respectively, wrote....
(B) Whenever there was a doubt (dns failure or suspected ip that wouldn't resolve), I let it have the real page.
(F) That, of course, is sound policy.
why not 404 it, or something similar?
The *one* cloaking page I have in place is basically a text version of all of the inner "description of the product" pages that search engines sometimes have a difficult time seeing because we use a database to put the pages together. It is basically a text-only page that links to the other pages of the site, and is something that a human being could go to and use just as any other page. I am hoping that even if an actual person from the search engines sees it, they will find the page does have content enough to leave it alone.
/me crosses her fingers.
Since I work at a big company, getting more content on the web site is a bureaucratical nightmare, so I am hoping this will suffice until we *can* get more content on the pages so we don't need to cloak.
Any comments are appreciated on this. I am very new to the SEO game and I am willing to learn from the veterans.
Hey Kim, thank you for stopping in. I've always enjoyed reading you and look forward to your new newsletter.
Air, that's true, it was mostly the moderators fault. Nothing like that will ever be allowed to happen here. If you ever think that it has, I want to know, and the problem will be corrected asap. This system is here 100% for the benefit of the users - there is no other reason for it's existence.
Anyway, returning to topic.
>why not 404 it, or something similar?
You can't afford the hit if is a spider you are dealing with - you have to give it something valid. It could just be link checking.
>re affiliate site banning.
This has me very concerned. While I don't do the affiliate thing myself, numerous customers are into the game in one degree or another. How long before the se's go after affiliates they don't care for? Does alta bury your page if it runs into a Yahoo or Hotbot search box? Have they done it already?
Hey thanks for fixing that ID up Brett.
As for the affiliate sites taking a beating there's no doubt. They likely have the biggest reliance on search engines than anyone. Not many people link to affiliate sites, directories shun them, and many just put up whatever the affiliate provides in banners, links, and likely content. A definite recipe for crashing and burning on a regular basis.
The one thing that is a little disturbing is that while many affiliate hubs are also being affected by this on AV, Amazon managed to get itself on the right side of AV. Anyone know how they managed this?
Getting buried for having competing content to a search engine's can't be far behind, and more than likely (although I don't have any evidence) it is happening already.
In a subtle way it has been happening for a long time, if you consider paid listings, and keyword targetted link and banner ads, as methods which to some degree "bury" your pages when they might stand in the way of SE revenue, then it has been happening for a while.