Log spam (referer spam)

Has anybody tracked down the wholesalers?

[robho]

Log spam has got to the point on one of my domains where it's beginning to be a real annoyance, tens of thousands of my database-driven pages per day.

Blocking the urls and the (mostly dialup/cable) IPs is a short term work around.

But it's clear (from the way there are a burst of varied referrer domains from the same IP in the same few seconds) that somebody is selling this as a "service".

I've started contacting the people behind the target domains where they appear to be mom-and-pop type stores (and where there are many hundreds of referrals from the site per day). One has told me he recently signed up for a "link ring". Hopefully I'll get more details from there and elsewhere and be able to track down the reseller(s), especially once these stores realise they've been conned by the "link ring".

I'm also considering moving on to the hosting companies where there's no response. I'm ignoring the adult domains as they're easier to filter by keyword (and are on spam-friendly organized crime hosts anyway).

Has anybody else taken this approach to referrer spam, or is any organization doing this? It really is at the point where I can't find the real users of the site buried in this junk.

[treeline]

Has anybody tracked down the wholesalers?

If they'd been tracked down and gotten what they deserved they wouldn't be doing it anymore.

[robho]

If they'd been tracked down and gotten what they deserved they wouldn't be doing it anymore

Tracking them down and stopping them are two very different things.

I'm setting up a redirect for the more persistent ones, to redirect them back to their own site (waste their bandwidth, not mine). Might make the clients notice.

[Dijkgraaf]

Actaully redirects might make them think that their referer spam is working when they see an increase in the number of hits on their page.
There is one persistant bugger comming from random IP's hitting a my sign guestbook page, I'll probably put a bit of logic in the issue a 403 Denied to them.

[BarryStCyr]

I'm sorry if this has been asked before, but what is referer spam or log spam? I check the glossary but it's not there.

Thanks
Barry

[robho]

what is referer spam or log spam?

It's when somebody accesses pages on your site with a fake addresss in the referrer string. Rather than showing the page they came from, it shows a page they want to promote.

This originated when some sites, especially blogs, automatically published lists of "top sites linking to us" and similar, or full online stats. The spammer's address would (falsely) appear on that list, creating a link for search engines and for people to visit their site. They couldn't think of any honest way to get visitors.

I presume most blogs would have stopped publishing referrers by now, and my sites (which are not blogs) never have published any form of stats. But that doesn't stop the spammers.

Why is it a problem? It creates fake traffic on your site, which uses a bit of server resources, and makes your page stats meaningless. It creates fake referrers in your logs, which mean it's harder to tell where your real traffic is coming from. It ups the number of visitors and distorts the "number of pages per visitor" stats.

In other words, it make the log files almost useless when done to excess. What's even more daft is that in most cases it doesn't even provide any benefit to the spammer.

P.S. I have eventually decided not to bother redirect the traffic back, or anything else really. Blocking the top dozen IP's and half a dozen "adult" keywords in the referrer has stopped 90% of them from accessing my database-driven pages, so that'll do, I don't have the time to follow it up.

[mikeyr]

but how did you block the top dozen or so IP's and keywords? I watch the logs and occassionally block ip's with iptables but it seems they get around within minutes and come from the same domain but different ip.

So I guess what I am asking is how to block ip's and domains and the adult keywords, I think I am doing it wrong.

[Dijkgraaf]

One bunch for referer spammers are also trying to POST to /cgi-bin/Formail.pl and varients, and strangely enough also two pages which have never supported POST.

[waziwazo]

I got similar strange porns "spams" link in my web log here a sample :

I did not post the whole thign, but the guy (actualy this must be a robot) used a dozen of different OS and web browsers all logging at same time

[edited by: jatar_k at 6:26 pm (utc) on July 28, 2005]
[edit reason] removed loglines - specifics and language [/edit]

[waziwazo]

Sorry about the "specifics and language" log lines
I should have read terms of services carefully before posting.

Only wanted to know if my "log spam" was similar to the ones robho have.

[EVOrange]

Robho, thanks for that clarification. I wasn't quite sure thier intent.

To repeat this question, how do you block by keyword?
htaccess, i assume, but could you post an example code line that i can tweak for my site?

Thanks.

EVO

[robho]

Sorry about the slow reply, I'm travelling.

I check the referrers within my php script (the entire site is script driven) and forward them to a tiny 403 page if it contains words which have recently appeared in log spam on the site, such as domains with "credit card", other (unprintable) words, or two hyphens in them. This saves them being a hit on the database and cuts the outbound traffic.

This could probably be done in a .htaccess but I've never tried that. I do also filter out a few fixed ips in the .htaccess.