Forum Moderators: martinibuster
The problem that I see is that these people are running adsense on these pages. I know that some of these sites are very possibly being built by other members here, but I just want to throw in my $0.02 and say that this is being a freeloader, and Google should discourage this practice.
For crying out loud, one page only had one link {mine} and about 70 words extracted from my article.
To make matters worse wathever script they are running creates the subdirectories to fully SEO their pages as in:
Disposable-domain.com/keyword/keyword/crappy-page-with-“borrowed”-content.html
This might make me some enemies, but really…. Isn’t this just a little evil?
I appreciate the links, I probably got a couple of dozen already, but I am willing to live without the couple of hundred referrals.
I said from the very start, bad is wrong. I was commenting on the fact that certain people in such a discussion project that "bad" onto lesser activities, and conclude the same "wrong". Uh uh.
From my posts do you get that I consider copying a million words or whatever *not* to be a copyright violation? If you answer yes, you need to re-read.
When the conversation compares snippets to millions of words, it is out of whack (as I noted).
You say scrapers grabbing a million words and using them to make a competing site and selling AdSense on it is wrong. No argument. Someone then says "yeah, I saw a site with nothing but snippets and AdSense. I agree, it is so wrong" -- well, those are very different things.
"snippets" may very well be fair use. Want to argue that? I won't bother -- it's well defined here in the US. Or you can stick to your personal defense. Whatever. But to mix them and conclude (infleunce others as well to conclude) that all snippets are bad and should be reported? Nope. I disagree.
As for this : If I am charitable enough to accept that some of your comments in msg 111 were tongue-in-cheek (which I don't accept, BTW) then all that's left in msg 111 is an attack on me. Is that what you intended?
I suggest you red your own feelings into that one. Tongue in cheek? Perhaps satire plus hyperbole is a better description. Oh, and I don't need nor appreciate any charity from you, thanks.
I won't address the other stuff... you don't get it about the risks of complaining, you have honourable competitors who don't play games behind the scenes, you hate sites less-honourable-than-yours rising in the SERPs just to sell ads (?), and on and on.
You remind me of the camera store owners of the early 90's here in the US. Internet retailers provided customers with full disclosure of all information, and a great price. Honourable luxury good retailers witheld information, sold at ridiculous markups, but offered an opportunity for the customer to touch the cameras and see them in real life before buying. So what did smart consumers do? Of course research on the Internet, visit the honourable shops just to touch the cameras in real life, and then bought them from the Internet. Poo poo on the honourable luxury rip-off retailers.... if they hadn't been ripping off the customers all along maybe they would still be in business today. Soooo short sighted!
You may have your honourable world, but welcome to the world of global competition. If you can't let your customers decide where to buy (or what site to visit, or what ad to click) that's your business. Me, well I'm thinking of going hunting for a new niche opportunity in the luxury goods market....
"snippets" may very well be fair use. Want to argue that? I won't bother -- it's well defined here in the US.
That's exactly my point. They are not DMCA violations. But scraping snippets containing identical keywords from different sites to make a million "KW1 KW2 KW3" pages is guilty of keyword spamming which is considered blackhat by Google, and I don't believe Adsense would allow it either.
If they do condone it, they risk alienating their audience, i.e. regular people looking for sites with the information they are after. And ultimately, people will begin switching to other SEs. Which is why Google/Adsense needs to act now to make sure such sites do not rank highly in the SERPS, or are supported financially through Adsense, as that will just encourage more and more of these useless pages.
However, if this is not the case, and I get a clear answer from Adsense that this is permitted, I'll gladly add a couple of million pages in as well, although I don't think Google would find this spamming acceptable.
Copyright infringments aside. I don't care if a "snippet", or a "million" words are being copied. Fact is, these scum sites are absolute trash that the internet doesn't need.
I don't give a rat's arse if some of you who are posting right here in this thread create these sites. I'll call it like it is, these sites are crap and you should be ashamed of yourself. But, like all crappy sites, they'll have their day before they are obliterated forever.
So please, feel free to attack my post because that seems to be the going thing in this thread because you all don't have anything else more substantial to add.
Dave.
You may have your honourable world, but welcome to the world of global competition
When the conversation compares snippets to millions of words, it is out of whack
"If someone blantantly violates fair use of someone's Intellectual Property by copying little snippets from each of their pages making a total reproduced content running into several hundreds of pages...."
That's not quite suggesting that nobody should ever use a single snippet from my site. We are talking hundreds of pages worth. I did not draw a conclusion based on "millions of words" and apply it to "snippets". C'mon, quit trying to twist things!
I'm trying to work out the logic behind your stoic defense of the scraper scum sites. You've stated that you have no "directory" sites. That is not of course an admission that you don't have any involvement in scraper scum sites. I've done a little digging around. You do seem to know a lot about the subject and the concept of scraper scum seems to be something very dear to your heart. You also seem very familiar with the legal issues relating to scraper scum sites. You're familiar with cases like the AA one where AA won under "trespass to chattels" and got a scraper to stop scraping their site. You've done a lot of research into the legalities behind scraping - I wonder why. Some of the things you've said about scraper scum sites in threads like this one [webmasterworld.com] and others are:
....so-called "scum" websites are often better than Google fpr producing relevant content and on-target ads. ....
I'd say the AdWords ads I have been clicking on have been extremely helpful to me.. I never would have found the genuine school comparison sites in all the SERP garbage if it were not for those directory sites running AdSense.
You've even made one detailed DIY post on how to build a scraper scum site (though you didn't use that terminology to describe it)
I suspect you have something to gain directly, and personally, from persuading people to not report scraper scum. I respect your knowledge, I'm bemused by what you'd call your "sense of humour", but, as long as I suspect you are directly involved in scraper scum sites, I question the credibility of your posts in threads like this one.
The part that I queation is You've even made one detailed DIY post on how to build a scraper scum site (though you didn't use that terminology to describe it)
I don't recall that one, but I wonder why you call it a DIY post but don't actually reference it. Perhaps that DIY commnet is a bit of a stretch, Macro?
The reason I say so is I know my position, and while I may have done that it would not be my usual style. Perhaps I was looking to elicit feedback.
Anyway since you are so interested, I don't participate in such directories either.
You observance of my interest in scraping is keen. The reason? That's where I earn my keep. As a scraper? Hardly. My area is competitive intelligence and competitive strategy. I am the guy who can save you from being killed by your competitors.... you know, the guys with 1% less ethics and 5% more knowledge than you, who have gone after your kitty. Many of them are so-called scrapers. I call them clever.
I am not expert by my own claim (I feel no one is) but I am aware. My clients desire to avoid being the victim of these drive bys and the like, and are working to prepared for the world ahead of us.
Theoretically, philosophically, and practically, what you call scraping has only just begun. It will increase many times over, very rapidly. It is unbeatable by complaining. It cannot be outlawed (except with some very, very bad laws with considerable side effects). It cannot be blocked per se. Expect protectionism early on... and a rapid ramp up of "excerpting" after that.
If webmasters don't wake up and address the real issues, web publishing will go the way of music publishing. Do we realy need another RIAA? Based on this thread and others on WebmasterWorld, I would expect Macro and EuropeFor Visitors to sign up right away, and bring a whole bunch of others with them.
Let's not delve into politics here.. you either agree with the RIAA or you don't. I don't, and I have a pretty clear vision of how RIAA spells doom for the music business (eventually). A similar boys club would be doom for web publishing as well.
When you boil it down it's about protectionism by and for the few so they can get really rich fast, on short term efforts that sell out the future for that short term personal gain (often in the name of "justice").
When you recognize that, you see that scraper's are doing the same thing, for the same reasons, and who's complaining? Of course it's the equivalent of the boys club.
Who's not complaining and doing something positive about it? My clients, to start. Keep your friends close... you know the phrase.
That's enough preaching.. as you note from my oldpaosts I don't try to preach - this time you expressed interest in my motivations.
Complaining and whining about scrapers is a bad move and helps no one -- it may even reinforce the "scrapers" (I remain convinced there are significant downsides to reporting more than the occasional website, and there are defintely downsides to showing your weaknesses, especailly in a forum known to be frequenetd by grey and black hatters).
As for stickies asking me about consulting (I have gotten several in the past), as others can attest I am not in the business of taking on work through WebmasterWorld.
[edited by: Jenstar at 11:51 pm (utc) on June 13, 2004]
[edit reason] as per member [/edit]
If anything our website creates a substantial market for the reviews. People visiting our website go to reviews that seem to be interesting based on the snippet we pick. If they like the site enough they are sure to visit again. If not, well at least that publisher got ad revenue from the non-return visitors as well.
I don't see how we'd be a useful service without publishing the snippet. Headlines alone can be incredibly deceptive or uninformative.
This surely falls under fair use under the DMCA. The only country in the world that I know of that doesn't even allow snippets or headlines to be used is Scotland. (So yes, we avoid posting Scottish reviews). So I certainly hope that people don't view us as "evil."
Do we realy need another RIAA? Based on this thread and others on WebmasterWorld, I would expect Macro and EuropeFor Visitors to sign up right away, and bring a whole bunch of others with them.
There are big differences between the record industry and Web publishing. With recorded music, the Web is used as a free (and illegal) distribution channel for a product that's sold on CDs. With Web publishing, the product (e.g., the text or other information) is already available for free online. So users derive no benefit from site-scraping of text as they do from illegal distribution of music files, and there's no built-in demand for the site-scraper's product.
As for the legality of site-scraping, that probably depends on how much is being scraped. A directory that grabs snippets of text isn't necessarily breaking the law--at least in the U.S., where the "fair use" doctrine allows reasonable excerpting for purposes of criticism, comment, news reporting, etc. For a discussion of fair use, see:
[publaw.com...]
Site-scraping of entire pages is not covered by fair use, and publishers who simply lift other people's work and re-use it can be sued. Such full-scale scraping can also result in exclusion from U.S.-based search engines like Google, Yahoo, and MSN if a DCMA complaint is filed--and we're now told that it can mean being expelled from the AdSense network, which is more to the point of this discussion.
If large-scale copyright infringement does become a problem, it will be a problem for Google, not just for publishers. Why? Because "content spam" will clutter Google's index in the same way that affiliate and e-comerce spam does now. That's why it's in Google's own interest to discourage content thievery by coming down hard on infringers in both Google Search and AdSense. But that isn't the same as expelling owners of directory sites that use "snippets" from the AdSense network, unless Google feels that the sites were created "specifically for the purpose of showing ads, whether or not the page content is relevant" (a use that's prohibited by the AdSense TOS).
Scraper scum can be removed from SERPS if SEs get their act together. There are some common features to the scraper scum sites. You've commented elsewhere on AI/heuristics; algorithimic blocking of scraper sites is not impossible. True, the sites will evolve and inject some autogenerated rubbish content designed to overcome new algos that detect scraper scum. And SEs will develop new means of detecting autogenerated rubbish ... .
I see no good reason to not file a complaint with Google and Adsense when a scraper scum steals hundreds/thousands/tens of thousands of your snippets. And I see no reason not to file a hundred complaints if you find a hundred such sites stealing your content. No self respecting SE wants rubbish. They want to remove it. They've provided easy routes for you to do so and from Jenstar's recent post [webmasterworld.com] Adsense terms have been modified to make very clear where Adsense stands on scraper scum sites displaying Adsense ads.
Your original hysterical rantings against filing complaints are now replaced with illogical arguments. But the end goal still seems to be to dissuade people from complaining. I'm not clear what intelligence you provide your clients but your posts are still attempting to scare people away from filing legitimate DMCA and Adsense violation complaints. You say it's a "bad move" and offers "significant downsides" but extend no elaboration to substantiate those wild claims.
But, we are getting somewhere :)
We've moved from:
In fact, at the start of this thread I was merely cautious and perhaps suspicious. Now I am making very very sure I do NOT GO ANYWHERE NEAR THAT FEEDBACK CLICK! Anybody else picking up the vibes? Am I too cynical/paranoid?
I remain convinced there are significant downsides to reporting more than the occasional website
asp4bunnies, read EFV's link. It's considered fair use when you use the copyrighted content for criticism and comment, parody and satire, scholarship and research, news reporting and teaching. Scraper scum sites don't of course qualify for fair use under any of those. They are using your content purely for their own commercial gain. Some of them give the impression that they are linking to you ... but they are not. They are thieves and should be treated as such.
I am the guy who can save you from being killed by your competitors
Whatever.
If you are seriously hurt by a scraper site, and the best avenue you see if to whine and complain, then I need not demonstrate any more. You are not prepared to be a successful competitive web publisher.
EFV says the web content is already available for free online... well I see ads on your site so it technically isn't available for free, is it? No more free than the CD of recorded music which I can easily copy (for free). If I argued in court that you put it up for free, you would flip-flop and claim it wasn't really free, but it generated revenue from ads and the like.
Your site serves ads and perhaps cookies and the like, and has a crafted path in many cases. Sure I can bypass them, just as I can bypass copyright on a music CD and make a copy. Today, the RIAA will go after me for copying the CD. The way you guys are headed, you will form an RIAA-like coalition and do the same thing (I can't access your web content unless I do it through your medium, whatever that is). We already see it with challenges to deep linking (attempts to force the user through the webmaster's choice of pathway) and challenges to spyblockers, "you may also like", and other innovations.
Re-publishing with ads and no additional value is wrong... I always ackowledged that. But how many of these "scraper sites" add value? To listen to you guys, none. That is not factual evidence.
Hyperbole can be a means to effective communication - especially when the argument is pathetic. To cite hyperbole as erroneous fact is, well, erroneous? Nonetheless it is a waste of time.
Macro says "Some of them give the impression that they are linking to you ... but they are not. They are thieves and should be treated as such. and so with such a categorization, Google is a thief for using their cache, their image browser, and a redirect system for it's own links to my content? That's right... they link to me and use my name and content for their own commercial gain, but pass the visitor through a redirect to block the sharing of that commercial potential with me. Rather than argue back about search engines being different (they are not different) this is the case with blogs, image archives, News, groups, etc etc etc... all very profitable commercial enterprises, and not all "worthy" of the sacred cloak called "search".
If you want to make bad laws be prepared to have them uniformly aplied.. even to you.
Finally, I said : I am the guy who can save you from being killed by your competitors and I meant that my clients hire me to alleviate the burden these kinds of competitors impose. If the burden was severe (as it often is) then my work does in fact "save them".
This that Macro wrote: Don't flatter yourself. You can't. If a business has a strong online and offline brand - and people actively search out the brand - the most you and/or any scraper scum site can do is control the route customers take to it. There's jack sh*t you can do to demean the brand itself. -- I have no idea what this is all about. It doesn't make any sense to me.
Not every business has a strong online/offline brand.
Not every online niche benefits from a strong brand, especially information brokers.
It is a rather trivial task to demean a brand (marketing 101).
Finally, here's an example. A family business several generations old in the health info space, had it's entire web complex scraped and re-posted with changes by a new competitor in the space. Instant competition. They lost over 40% of their business in the first 6 months. They held onto SERP #1 but new competitor was #3 and #4 everywhere they looked. Should they complain to Google? to the FBI?
No. They approached me and we chose to compete, and have won. The company has a new understanding of competition, and a new appreciation for online marketing. They were competitive only due to their dominant position - and that doesn't survive in today's world. Now they are competitive in the online space as well. Market share is back up to 68% of previous levels, and rumor has it "new competitor" has now chosen a niche for himself that doesn't directly threaten the family business. But they understand there could be another one tomorrow, so they are preparing. I know he only chose another niche because we challenges his assault.
As you should. Whiners will lose.
I'm sure it's been mentioned, but it bears repeating - this is precisely what Google (and any other SERP) does. They crawl, scrape, quote, and profit off of "AdSense" money.
Google, of course, has "cached" and "images" to make matters worse.
The question should be then - is Google evil?
I need to repeat what other people have said. It's so obvious and really really bears repeating.
1. Intent: Search engines want to send me traffic
Clearly false. They want people to compare your site with their AdWords (overture ads, whatever) customers. Please please don't for a second think that search engines are a public service.
2. Method: Search engines trawl my site, pick up pages and rank them based on how relevent they think my content is.
Again, you are assigning motives that you can not compellingly know exist. Did yahoo start up sitematch? I don't even want to talk about the motives behind the people at Looksmart.
3. Control: I am the one who controls the pages the SEs and reputable directories show from my site.
This is a very very good point. However, who says all scum sites don't make available an opportunity to keep them out via robots.txt. Even if they did, it would only take away a small percentage of their content. I don't think this will solve your problem .. however, I do agree, if they do not respect robots.txt, they are 'sewerscum'.
4. Content: The content of SEs is the quality of their results/their algo/their search features.
The algo in this case is the human mind. The human mind is often better than any AI program.
I think you just need to be patient. Google doesn't like competitors. They will find away to keep these sites out of their directories.
But make no mistake - these people are valid competitors to Google. They are not 'sewerscum'.
Having participated in a court case involving spidering a site, this actually was the defining argument the defense made.
It was ok to crawl and scrape our website because the robots.txt allowed for it.
Some people say that this is unfair, because they were not aware of robots.txt .. however, ignorance is not an excuse.
How can you let Google crawl you but nobody else? Yahoo? LookSmart? Msn? Where does it stop?
You have to specifically say or accept the consequences.
I fully agree though, if you do not respect the robots.txt you are 'sewerscum' / 'evil' / whatever it is that Macro said. And, yes, robots.txt should apply to 'human copying' as well.
In fact there is probably legal precedence for this, check out [auction...] bytes/cab/abn/y04/m04/i16/s03 (remove the space, as it is getting filtered by WebmasterWorld)
and I just saw on WebmasterWorld somewhere (yesteray?) a post that quoted someone at Google saying high-maintenance publishers would likely be dropped. Doesn't that sound like frequent complaining is a bad idea?
Depends what the publisher is complaining or inquiring about. The publisher (especially one with modest revenues) who's constantly writing Google to ask, "Can I do this or that?" or "When will my report be updated?" or "How do I know you guys aren't cheating me?" is likely to wear out his welcome pretty quickly, because he's generating more overhead expense than profit.
Reporting activities that are illegal, or that compromise the integrity of Google's search results or AdSense network, are a different matter altogether, because such reports ultimately benefit Google. (IMHO, the frequency of such complaints by an individual publisher isn't likely to be great enough to attract notice in any case, unless the publisher has made it his hobby to track down and report offenders on a regular basis.)
In the early days I emailed G around 5 - 10 times a week and my CTR was very bad and revenues was way down. I know of several websites that make only around a 100-200 bucks a month and they keep the G support-team very busy.
If G has a shortage of support teams they can always higher more staffs.
If an account however gets disabled I would first look into manipulated clicks and I have that strong feeling that around 80% (could be rubbish too from my site) of suspended sites deserve it and it is in the inteerst of the Adsense publishers and the advertisers to get those bad sites kicked off the system.
Neither do threats...
I'm thinking of going hunting for a new niche opportunity in the luxury goods market (in reply to me saying that I was in that market?! I'm trembling in my boots at the very prospect;))
You also contributed numerous other insinuations to stifle debate, warnings about black hat webmasters here and about not disclosing too much (designed to scare those who have websites in their profiles... designed to create insecurity among those who have something to complain about), warned that one shouldn't complain because it involves clicking a js(!), and even made a suggestion that reporting the scum will strengthen their position!
Re-publishing with ads and no additional value is wrong...
I know of several websites that make only around a 100-200 bucks a month and they keep the G support-team very busy.
In any case, legality isn't necessarily the issue here. Google could easily decide that, for quality reasons alone, automatically generated sitescraper directory pages don't belong in the AdSense network or Google's SERPs. If such sites are left unchecked, "content spam" could generate as much duplicate-content clutter in Google's SERPs as boilerplate affiliate and e-commerce pages do now. It's in Google's own interest to keep the sitescrapers from taking over the AdSense network and its SERPs. That's why I wouldn't be at all surprised to see "automatically generated directory pages" added to the list of forbidden AdSense page types.
As for the DMCA, it's definitely worth using in cases of clearcut infringement (such as the lifting of entire pages, articles, or photos).
In any case, legality isn't necessarily the issue here. Google could easily decide that, for quality reasons alone, automatically generated sitescraper directory pages don't belong in the AdSense network or Google's SERPs.
I agree 100%. This is exactly what I'm hoping for.
Legal recourse of some sort is likely to be required. My suspicion is that this is why Google set up the DMCA route with AdSense.
AdSense is irrelevant. This can as easily be done with affiliate ads and appropiate contextual technology.
That may be true for mass-market keyphrases, but there are millions of keyphrases and keywords that don't have related affiliate programs or contextual ads from other sources.
As for whether Google can keep this out of the SERPS remains yet to be seen.
Maybe, but encouraging "sitescraping" by paying AdSense revenue to sitescrapers is likely to make the SERP-clutter problem even worse than it would be otherwise.
but encouraging "sitescraping" by paying AdSense revenue to sitescrapers is likely to make the SERP-clutter problem even worse than it would be otherwise.
But the number of terms and the frequency with which these sites are showing up in Google SERPS seems to be increasing. Other big search engines seem to be doing a better job of keeping them out of SERPS. I wonder what Google has up its sleeve.
Legal recourse of some sort is likely to be required. My suspicion is that this is why Google set up the DMCA route with AdSense.
Legal recourse is not needed to keep these sites out of Adsense or the SERPS. It is a matter of quality of the network. I got a theory about the mentioning of the DMCA in the FAQ, and it has to do with a lawyer in the GooglePlex recommending it. This is fine with me since that is what lawyers do.... They protect you from other lawyers.
Nevertheless, I seriously believe that these sites can be removed solely on the basis of quality of sites based solely in Google's discretion. I also believe that Google's reaction to these sites will be an indication of how they lean in the evilness scale.
P.S: I also agree with EFV that leaving these sites in the network is encouraging cluttering.
So before you go reporting sites for "violations", you should make very sure that Google considers them violators. Perhaps, instead of an accusatory email, someone should send Google a representative URL and ask if they consider it a legit site/page for Adsense.
If they say no, report away. If they say it's OK, it might be best to back off.
send Google a representative URL and ask if they consider it a legit site/page for Adsense
RE: that scalp you claim as a prize:
1. It's an automated script.
2. The domain name was not used for it's branding value... it was probably disposable.
3. I bet it's back online somewhere else in a day.
You may feel better, Macro, but was it worth aggravating Google over something they are working on automatically handling in the future?
was it worth aggravating GoogleGoogle didn't seem aggravated:). They were very polite and thanked me. I don't believe that the reading of my brief feedback (including the optional email address bit which I filled in) kept them busy for very long. I don't believe that sending me a canned email was a full time job for the plex ;)
So far Google seem appreciative of our reports and seem to agree that these reports help them improve the program.
I don't get it... how could any webmaster be unhappy about getting a valid HTML link to their web site?
Links to our sites is a good thing, right?
If you're like me in fact, you likely spend a good amount of time trying to get people to send a link in our direction...
Directories, search engines and meta search engines are doing us favors by including our links. We should consider ourselves lucky. I do.
It's only when that minority of bad seeds REMOVE the links or plain steal content (hijack, redirect, cloak etc.) that makes me mad, but if you guys are getting off on bad mouthing directories for "stealing" your content, then you better send the same amount of hostility towards Google, Yahoo, Alta Vista etc. as they do exactly what you are complaining about.
Links are what the net is all about and if you were a smart webmaster you'd realize that directories and search engines allegedly "stealing your content" and linking to you are contributing PageRank to your web sites. Not a bad thing in my book. They are heloping you become more popular.
I'm happy when I surf Google and find directories and search engines linking to me... only when I see the shady practice of removing or disabling the links do I get mad.
Otherwise I consider it a gift. Maybe you should too.
This reminds me of how preposterous that one newspaper was for suing becuse it wanted links to point at it's index page and not directly to th article pages. What a waste of time... count your blessings that anyone links to you period!
Have a rice day!
Mick
