Welcome to WebmasterWorld Guest from 54.227.68.206

Forum Moderators: open

Message Too Old, No Replies

Is there a way to block single keyword referrals from Live Search?

     
10:23 am on Feb 17, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member marcia is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Sept 29, 2000
posts:12095
votes: 0


There's no way that these "visitors" are arriving by finding the site(s) with those single word searches, and I have a strong suspicion that it would be best to block them altogether.

Is there a way to block those altogether from accessing a site? Or at least not going any further than the first page they hit?

6:46 pm on Feb 17, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 22, 2001
posts:2450
votes: 0


You could always do some scripting to check for a microsoft live reference in the HTTP_REFERER header.

This could be done via mod_rewrite, and send such hits over to a script for further processing, such as adding their IP to a 403 ban list, etc.

7:16 am on Feb 18, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member marcia is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Sept 29, 2000
posts: 12095
votes: 0


I don't want all from Live banned, just the ones coming in on one word searches, because they're scrapers.
8:32 am on Feb 18, 2008 (gmt 0)

Preferred Member

10+ Year Member

joined:Nov 2, 2005
posts:505
votes: 0


I use this (in .htaccess):
RewriteCond %{HTTP_REFERER} (milf) [NC,OR]
RewriteCond %{HTTP_REFERER} (x x x) [NC,OR]
RewriteRule .* - [F]

to block the referral spammers, which will work for you too if all of your people are sending referrers.

If I am not mistaken, I got this code a while back off webmasterworld. :)

10:35 am on Feb 18, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member marcia is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Sept 29, 2000
posts:12095
votes: 0


>>if all of your people are sending referrers

They aren't all unwelcome and not all are a PITA. Most visitors are 100% legitimate and more than welcome. If someone comes in from search using a search phrase such as "discount yellow widgets" or "yellow widget stores," then it's a living, breathing human being and they'll find what they're looking for. But if they arrive with the single search word "widgets" then it's software running for either data mining (no biggie) or to scrape for MFA sites, and the latter are a nuisance and represent a hazard for some sites if they go far enough, which many do.

I do know there's a way to sniff for s/e referral query strings and then process accordingly, but it was a long time ago, so my memory on it is very dim.

>>If I am not mistaken, I got this code a while back off webmasterworld.

Jake, I've been spending a lot of time "reading" in the code forums at WebmasterWorld (which I heartily recommend, rather than just copying and pasting), and unless I can "read" what something is doing, I won't use it until I know what it'll do.

RewriteCond %{HTTP_REFERER} (milf) [NC,OR]
RewriteCond %{HTTP_REFERER} (x x x) [NC,OR]
RewriteRule .* - [F]

Do (milf) and (x x x) represent what actual query words could be? If so, then that could do it.

[edited by: Marcia at 10:49 am (utc) on Feb. 18, 2008]

11:26 am on Feb 18, 2008 (gmt 0)

Preferred Member

10+ Year Member

joined:Nov 2, 2005
posts:505
votes: 0


milf and x x x are actual words I filter, the x's of course need to remove the spaces because WebmasterWorld filter the "word"..

And I do not copy/paste anything until I believe it won't break my site either. Only posted the reference as a working, live example that already works. :)

I use it myself, which is why I mentioned "I got this code a while back off webmasterworld" -- not to claim this as my work.

Believe me, I do not spend my time looking for ways to make myself look smart by answering questions I don't understand.

By the way you described your problem.. the remedy sounded as simple as the htaccess filter I use to block spam bots, that is the only reason I posted it.

11:34 am on Feb 18, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member marcia is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Sept 29, 2000
posts:12095
votes: 0


By the way you described your problem.. the remedy sounded as simple as the htaccess filter I use to block spam bots, that is the only reason I posted it.

If those are the actual query words, then that might just do the trick. I've got a fairly new domain not doing much of anything on other engines yet that gets those requests constantly, it sure couldn't hurt to try.

Thanks, will do!