|Scale of fraud, or...|
| 8:53 pm on Aug 5, 2003 (gmt 0)|
I'm wondering if Google's AdSense network is experiencing massive levels of suspected fraudulent clicking, or if its fraud detector has gone bonkers in the last 24 hours or so.
Yesterday afternoon, I got a "fraudulent clicks have been detected" e-mail from Google. It was the same boilerplate e-mail that other publishers have received. I wrote back and got a polite if generic reply.
This afternoon, almost exactly 24 hours later, I got another "fraudulent clicks have been detected" e-mail. Here's the interesting part: from the case numbers in the message headers, it would appear that nearly 15,000 such e-mails have been sent out in the last 24 hours.
If that figure of 15,000 "fraudulent clicks" e-mails in 24 hours is is correct, the incidence of suspected fraud is nothing short of mind-boggling. It also suggests that Google's "fraudulent clicks" e-mails are generated automatically, since it's hard to conceive of a staff large enough to review and handle reports of fraudulent clicks manually at the rate of nearly 600 per hour on a 24-hour basis or nearly 1,800 per hour in an 8-hour business day.
It's possible, of course, that those case numbers reflect fraudulent clicks detected for the AdWords program in general, and not just for the AdSense network. Either way, Google's fraud detector is sounding the alarm nearly 15,000 times per day, which suggests that fraudulent clicks may be nearly as big a challenge for Google as search-engine spam.
Side note: I haven't done any fraudulent clicking (in fact, I've never clicked on even one AdSense ad), so the clicks could have been generated by anyone. I don't see any obvious anomalies in my AdSense statistics (probably because of the "smoothing effect" of some 3,500 pages with a wide range of AdSense ads). However, in analyzing my logfiles for the past several days, I saw an unusually high number of hits from one or two visitors each on three domains that I've never heard of before. I've reported these to Google (in fact, I've offered to forward my logfiles), since I'm as interested in deterring fraud as Google is.
| 9:05 pm on Aug 5, 2003 (gmt 0)|
Case numbers might cover every single matter including fraud detection. However, I would have to agree that everything seems to be automated and sometimes the intervention of a real person is needed.
I just don't get what kind of activity would raise a red flag on a site like yours. Compared to mine is like 35 times as big with 50 or 100 times more traffic. It must detect number of clicks from one ip? clicks from people browsing from known proxies?
I sure hope that you either stop getting the emails or that you get someone to look at your case.
| 9:19 pm on Aug 5, 2003 (gmt 0)|
That is a very interesting observation. If you have gone through the "contact AdSense" route for asking a question, does the number from that also have a case number that would appear to be in the same numbering pattern as the fraudulent clicks/impressions email? With all the problems with new pages not showing targeted ads over the past few days, I am sure there were plenty of people contacting AdSense over it.
If the numbers don't match at all, WOW. I can't imagine how they could process them to give honest publishers a fair shot at proving their innocence if AdSense is handling a case load of 15,000 per day.
It also gives an insight into just how many publishers are using AdSense now, too.
Nice find, europeforvisitors!
| 9:40 pm on Aug 5, 2003 (gmt 0)|
>>If that figure of 15,000 "fraudulent clicks" e-mails in 24 hours is is correct
I would be careful about reading into "invoice numbers" too much. Many companies these days do play around with numbers in order to avoid people seeing, for example, how big their business is.
You can try to check a market by buying online from a competitor, getting an invoice with a number, then buying again a few weeks later, and seeing how many invoices they have sent in that period.
Many companies try to get round that by messing with their numbers to stop you seeing what is going on.
On balance, I agree with you that it looks as if they are sending out large numbers of these automatic emails. Presumably, if it is that large a problem, they start with the biggest earners and work down.
Confirms my belief that Google will have to reduce the number of content sites, if for no other reason, than to make the control of fraud manageable
| 10:09 pm on Aug 5, 2003 (gmt 0)|
|If you have gone through the "contact AdSense" route for asking a question, does the number from that also have a case number that would appear to be in the same numbering pattern as the fraudulent clicks/impressions email? |
Thanks for asking--I hadn't thought of that. :-) I just went back and looked at the one numbered e-mail that I received from AdSense support back on June 19, when I was accepted into the program after initially being rejected, and it's 582,051 lower than the number of the latest e-mail from email@example.com.
|I would be careful about reading into "invoice numbers" too much. Many companies these days do play around with numbers in order to avoid people seeing, for example, how big their business is. |
That's certainly possible, although the numbers in my three communiqués from Google (one from support, one from adsense-spam) have been higher each time. It's also possible that each Google employee is using a different stack of numbered forms. :-)
|Confirms my belief that Google will have to reduce the number of content sites, if for no other reason, than to make the control of fraud manageable. |
I think that's a legitimate point, although advertiser fraud may be an even bigger problem than publisher fraud--in which case reducing the number of content sites (or raising the minimum payout to achieve the same result) may not have much effect. If the owner of Bud's Booking Service is clicking on ads for Harry's Hotel Reservations for competitive reasons, there isn't much that the publisher of a content site can do about it.
|I just don't get what kind of activity would raise a red flag on a site like yours. Compared to mine is like 35 times as big with 50 or 100 times more traffic. It must detect number of clicks from one ip? clicks from people browsing from known proxies? |
I've wondered about that, too. My daily CTR doesn't vary a great deal, and I haven't seen any obvious spikes in my AdSense numbers. So it seems likely that any fraudulent clicking has been on specific ads, where a relatively small number of clicks could trigger an alarm without affecting my overall CTR, total clicks, or revenues.
I've also wondered if Google's fraud detector mightn't detect what appear to be anomalies but aren't. Example: I often see ads for a certain river-cruising company on my barge and river cruises pages (which represent less than 1% of my site's content), but the number of clicks for those ads must be quite small. In the last couple of days, however, I've mentioned the name of the company in a news blurb on my home page, and all of a sudden I'm seeing ads for the company on my home page and on the news story itself (which has been attracting a fair amount of traffic). An automated detector might see a sudden spike in clickthroughs for that advertiser and think that something was amiss, when in fact the additional clicks were the result of editorial changes and Google's own targeting.
This whole situation is a bit annoying, and it makes me glad that I don't have all my eggs (or even a majority of my eggs) in the AdSense basket. AdSense is a great concept, but--as I stated in another thread--the program's greatest weakness may be an overreliance on algorithms at the expense of human judgment.
| 11:09 pm on Aug 5, 2003 (gmt 0)|
There will probably be some kind of checksum in any tracking number system, so even if the numbers are steadily utilisation might be around 10%.
| 11:12 pm on Aug 5, 2003 (gmt 0)|
A lot of traffic to a single page isn't enough to set of the fraud detector. On only my second day running AdSense, one of my pages was Slashdotted -- 5000+ impressions instead of a few score -- with no sign of any problem.
* I only get my site stats daily, so I noticed this in AdSense first!
| 12:37 am on Aug 6, 2003 (gmt 0)|
|A lot of traffic to a single page isn't enough to set of the fraud detector. |
I was thinking more along the lines of an increase in clicks on certain ads. In other words, if normally gets three clicks a day from mysite.com and suddenly gets 20, Google's fraudbot might sit up and take notice.
| 12:50 am on Aug 6, 2003 (gmt 0)|
|In other words, if normally gets three clicks a day from mysite.com and suddenly gets 20, Google's fraudbot might sit up and take notice. |
I do not think that just multiple same ads clicking would trigger the red-flag. Google should have some other additional detecting criteria, such as all clicks from single IP/range of IPs in limited time period, etc.
Also, I do not feel it would be a "massive scale of fraud, ..." and probably it would just be a single case because I have not seen any other publishers/webmasters claiming/shouting it.
Finally, the ticket numbers provide little clue of Google's operation. They should have some kind of "CRM" system to communicate/respond/track emails, covering all Google's segments/programs, including our Froogle stuffs!
| 2:48 am on Aug 6, 2003 (gmt 0)|
|I do not think that just multiple same ads clicking would trigger the red-flag. Google should have some other additional detecting criteria, such as all clicks from single IP/range of IPs in limited time period, etc. |
Some publishers have reported getting dumped or receiving strong warnings for alleged fraudulent clicks, while others (including yours truly) have merely received vague warnings. I wonder if this means that:
1) Clicks from the publisher's IP address or IP range are viewed as de facto evidence of guilt, while...
2) Clicks from a different IP address or IP range are viewed as being inconclusive, since they could be coming from a user on the other side of the world or from a proxy used by the publisher (e.g., from the clickbot software that was discussed here recently).
In the first case, Google might not hesitate to dump the publisher; in the second example, Google might tread more carefully, especially if the fraudulent click revenues were modest in the overall scheme of things and good judgment would argue against the publisher's being at fault.
The obvious question is why Google would send out warning e-mails automatically when there's no evidence to suggest that the publisher is at fault. My guess: Google figures that, if the publisher is the fraudulent clicker, the publisher may think twice before sinning again--and if not, it never hurts to send a warning. IMHO, this is a pretty heavy-handed approach, and it would make more sense for Google to automatically check the offending IP address against other known addresses (including those of advertisers) behind the scenes. Publisher fraud undoubtedly exists, but there certainly are many other people (including advertisers) who have equally strong motives for fraudulent clicking on AdWords and AdSense ads.
| 3:00 am on Aug 6, 2003 (gmt 0)|
|The obvious question is why Google would send out warning e-mails automatically when there's no evidence to suggest that the publisher is at fault. My guess: Google figures that, if the publisher is the fraudulent clicker, the publisher may think twice before sinning again--and if not, it never hurts to send a warning. IMHO, this is a pretty heavy-handed approach, and it would make more sense for Google to automatically check the offending IP address against other known addresses (including those of advertisers) behind the scenes. |
I think this would not be an easy issue for everyone, including Google.
Probably fewer "stupid" publishers/webmasters would conduct "fraudulent clicks" from fixed IPs with their computers. How could Google tell it would be made by the publisher or the third party?
I just have no idea. Maybe someone is laughing at somewhere!
| 6:35 am on Aug 6, 2003 (gmt 0)|
|Probably fewer "stupid" publishers/webmasters would conduct "fraudulent clicks" from fixed IPs with their computers. How could Google tell it would be made by the publisher or the third party? |
Not everyone is technically savvy. And people such as advertisers' competitors or troublemakers may not bother to conceal their tracks, since they have nothing to lose unless they're AdWords advertisers or AdSense publishers.
| 2:56 pm on Aug 6, 2003 (gmt 0)|
|Not everyone is technically savvy. And people such as advertisers' competitors or troublemakers may not bother to conceal their tracks, since they have nothing to lose unless they're AdWords advertisers or AdSense publishers. |
It is wrong that Google dumps publishers who have not commited anything, but for example their competitor , or somebody who doesn't like their content, did it. It is utterly Wrong.
Why Google cannot introduce some technical solution for this problem of fraudulant clicks? For example: after more than 3 cliks from the same IP Google should not charge the Advertisers and should not reward the publisher... I am certain that some technical solution for this problem is possible.
However current situation is utterly wrong and unfair.
| 3:18 pm on Aug 6, 2003 (gmt 0)|
|However current situation is utterly wrong and unfair. |
I will agree that it is unfair, but I think that too much burden is placed on Google if one expects them to act as a police force and do a lot of investigating or setting up controls on the controls. I am not saying that doing so would not be beneficial in the long run, but I would need to know how expensive it would be implementing them.
I look at it from the point of view of whether or not it is worth it for Google to keep all the advertisers in the network. Lets say that so far Adsense has 20,000 publishers that are serving 400 million impressions a day. Of these there are about 500 publishers that serve 10 million impressions a day that somehow get flagged for abuse. You could either let the automated system deal with it and probably hire a couple of people to answer emails from people responding to the warning emails for a cost of $100,000 a year or hire 10 people to do a lot of investigating and spend $500,000 a year in salaries alone.
It all comes down to money. In addition, the TOS to which we agreed to gives Google to do pretty much whichever it pleases…. Hmmm, now that I think about it some, I take back what I said about agreeing that this is unfair. Frankly, it is not unfair since we agreed to let Google interpret and have a final say. After all, it is their program and they get to run it the way they want to run it. I would think otherwise if this company had a history of cheating publishers such as some of the lesser known ad networks, but I am sure that the number of people accused of fraud is the result of either true fraud or of the fraud filters picking up on something and interpreting the wrong way.
In other words, there is no big conspiracy here to defraud. There might be a case that Google has made a decision not to invest in the refining of fraudbot, but hopefully this situation will be reversed as the program matures
| 4:08 pm on Aug 6, 2003 (gmt 0)|
Post removed by Tiebreaker as a precaution
My thanks to Cornwall for alerting me :-)
| 8:37 pm on Aug 6, 2003 (gmt 0)|
Look at it this way:
Suppose Google has a system for detecting fraud based on the recorded pattern of clicks and impressions. Suppose all publishers are routinely monitored by this system.
Further, suppose the system is 90% accurate. This means that:
1. If you committed fraud, then the system has a 90% chance of catching you.
2. Conversely, if you are innocent, then the system correctly identifies you as innocent in 90% of the cases; there is a 10% chance that you will be flagged even though you are innocent (false positive).
Let's assume 90% of publishers in AdSense are honest, while 10% are fraudsters.
Suppose Europeforvisitors has been flagged by the system as having committed fraud. What is the probability that Europeforvisitors is actually a fraudster?
ANSWER: 50%. In any group of 100 publishers accused of fraud, only 50 are actually fraudsters.
Tomasz P. Szynalski
| 8:59 pm on Aug 6, 2003 (gmt 0)|
> Why Google cannot introduce some technical solution for this problem of fraudulant clicks?
Seems like a complex problem. The industry I am in suffers constant hitbots attacks. Some hitbot programs are very sophisticated, will go several layers deep, run proxy lists and time the pace of click-throughs so that they appear as if done by real people.
you may have just had some residual hitbot traffic that triggered G alerts. We were recently terminated from a program because of allegations of click fraud when ALL our data pointed at 3rd party hitbot traffic.
There was nothing we could do.
| 9:18 pm on Aug 6, 2003 (gmt 0)|
|you may have just had some residual hitbot traffic that triggered G alerts. We were recently terminated from a program because of allegations of click fraud when ALL our data pointed at 3rd party hitbot traffic. There was nothing we could do. |
This sounds like a potentially severe problem not just for publishers, but also for AdWords/AdSense and other PPC networks. After all, there are reasons why people might use hitbots besides lining their own pockets. An advertiser, for example, might want to drive up another advertiser's costs or make it harder for the competing advertiser to run an effective campaign. If Google dumped every publisher that was the victim of hitbot attacks, it would have trouble maintaining a profitable AdSense network--and at the same time, it would be experiencing the same hitbot problems with the AdWords on its own SERPs.
Maybe the real future of AdSense and AdWords isn't in CPC, but in CPM advertising. (Hitbots could suck up CPM impressions, too, but only by massive attacks that would be relatively easy to spot.)
| 7:33 pm on Aug 7, 2003 (gmt 0)|
post removed by poster.
[edited by: TravelMan at 9:03 pm (utc) on Aug. 7, 2003]
| 8:17 pm on Aug 7, 2003 (gmt 0)|
|Their reply tells me that they have reviewed my situation thoroughly and accurately, and have |
determined that my account has violated their program policies.
For what it's worth, after 5 weeks in AdSense I received a fraudulent clicks notification last night. I replied asking for details, offering to cooperate and explaining that we haven't done anything fraudulent and haven't enticed users to click ads. I received a reply which starts "Thank you for the update. As you know, Google treats instances of fraudulent click spam very seriously. However, we understand that the recent activity in question may be due to circumstances beyond your control."
It then went on to point out how to check ad URLs without clicking them and pointed me to the terms and policies even though my previous reply to them stated I had read and understood the terms, policies and FAQ and believed we were in compliance.
I did mention a few things which I thought may have caused my site to be flagged, but they ignored those. Here is some background and details about AdSense on my site and communication with Google. Maybe some will find it informative.
The site is a crime information/content site. For the first several weeks many of the pages showed targeted ads. On July 12th many pages previously showing paid ads began showing PSAs, CTR which had always averaged below 1% dropped significantly and stayed low, as did the CPC which was never high and dropped to close to what I believe the minimum is.
I contacted Google and they explained that the pages in question (I never mentioned specific pages) "contains negative content" and "we only serve family-safe ads". I think they didn't take a good luck at the site because though many of the pages contain words like rape, molestation and porn that's simply due to the type of crimes that are the focus of the site. The ads were a good match for the site, the users and the advertisers and I'm sure the site was exactly the type of site the advertisers would want to advertise on.
I mentioned this, but did not get a reply until yesterday, nearly 3 weeks later. Here's part of their reply. "We have recently upgraded our content algorithms to improve the effectiveness of AdWords ads for both our publishers as well as our advertisers. While our policies may have initially allowed you to run ads on this site, this recent
upgrade may impact what ads are displayed on your website."
They're true to their word. This morning I noticed most of the ads on certain pages that had been well-targeted were now showing ads that had never appeared before and not only have nothing to do with the page displaying them, but nothing to do with any pages on the site.
They also said "The general topic of your site will cause this to happen on many of your pages. If you do not wish to receive untargeted ads on these pages, it is recommended that you remove your code from these pages."
I realize there are some topics that will cause their algorithms trouble and I'm not surprised mine does, though a human would clearly realize that a crime site for victims, survivors, past abuser and their relatives, friends and neighbors could very well have words that are found on sites that aren't "family-friendly".
A week later a significant news event occurred which indirectly caused page views and impressions to increase significantly due to legitimate visits increasing dramatically. CPC stayed low and CTR actually decreased.
Yesterday night, 4 hours after the Google Team answered my email from 3 weeks earlier I received a fraudulent clicks notification. Since I'm only privy to the AdSense reports, CPC and CTR remain low (CTR below 1% for 27 straight days) and Google won't provide me any clues so I can at least attempt to understand why they thought there was fraudulent activity and what it was, I feel helpless. I've removed AdSense from most of the site to reduce the chance the site gets booted before I get the first payment since I'd hate for everything earned to date to be withheld.
Of course I'm not pleased with the related searches and ugly blank search boxes either.
I did point something else out which they ignored and which I contacted Google technical support separately about and am waiting to hear back about. On August 2nd a database error led to much of the site showing a PHP database connection error instead of the desired content. Mediapartners apparently visited some of the pages because they currently display ads for database related products and services. Obviously this was unintentional and since my CTR has been below 1% with targeted ads I don't think having totally unrelated database products/services ads would increase CPM so such a "strategy" would be naive and silly. In any case, that's my best guess to the trigger for the fraudulent click notification, especially since traffic and impressions have come down steadily since they peaked 2 weeks ago and I received no such notification then.
| 6:11 pm on Aug 8, 2003 (gmt 0)|
As a follow-up to my previous post, the Google tech support team emailed me today to let me know that my site shouldn't be showing the database product/services ads anymore (I verified it). They also said that they contacted the fraud investigation team and that I am not being held responsible for the instances of recent fraudulent clicks.
I still was never provided information about what the fraudulent clicks are, but if I connect the dots I assume that my suspicion that they were related to these ads is probably true.
In any case, this is evidence (anecdotal maybe, but evidence none the less)that Google doesn't just boot everyone whose site triggers their fraudulent activity notices.
| 9:25 pm on Aug 8, 2003 (gmt 0)|
I contacted Google and they explained that the pages in question (I never mentioned specific pages) "contains negative content"
richmondsteve, your story is charming and moved my heart. Thank you for it, man.
just one remark: maybe Googliness should introduce a list of what is good and what is evil ("negative content") because I have a suspicion that soon any page that mentions words "Yahoo" or "Microsoft" or "MSN" more than 5 times - competitors - will be marked as "negative content" and by means of "fraudulant clicks accussation" publishers will be dumped without mercy.
I still was never provided information about what the fraudulent clicks are, but if I connect the dots I assume that my suspicion that they were related to these ads is probably true.
You lost me. Maybe I am not intelligent enough to grasp it. Please elaborate. What do you mean by "these ads" above? Your story is nice but a bit too complicated (remember: KISS - Keep It Simple & St~pid)
| 10:09 pm on Aug 8, 2003 (gmt 0)|
|...maybe Googliness should introduce a list of what is good and what is evil ("negative content")... |
It's probably a combo of words, frequency and density. My problem is that on my information and community site for sex crime survivors and others words like rape, porn, sex and molestation are prominent and apparently cause the site to be flagged for negative content. I think it's safe to assume they're trying to flag adult content sites so I'm optimistic that they'll eventually tweak it to exclude sites like mine or simply manually flag mine to be overlooked for such tests.
|What do you mean by "these ads" above? |
PolishGuy, considering the length of my post I'm not surprised it's not clear! On August 2nd, a corrupted MySQL database caused some PHP pages to display a PHP generated MySQL database connection error message instead of the content that was normally displayed. Mediapartners visited some of these pages and decided they were about MySQL and databases.
As a result, from August 2nd until last night or this morning those pages showed ads for software, products and services related to MySQL and other databases. This was despite the fact that I corrected the MySQL issue the same day (August 2nd) and it had been showing the proper content from then forward.
On the 6th I got the fraudulent click notification. I wasn't given details. I contacted Google tech support via their online form and informed them about the mis-targeted ads and they said that they contacted the fraud investigation team and that I am not being held responsible for the instances of recent fraudulent clicks.
They didn't actually say that the "fraudulent clicks" were clicks of the database related ads, but unless they were their response doesn't make sense. And I was never told that there was a high CTR or anything else most here think would trigger the notification. I suspect in my case it may have simply been a script or human detecting an apparent attempt to force AdSense to display specific types of ads - in this case ads that had nothing to do with the actual content of the pages' normal text.