homepage Welcome to WebmasterWorld Guest from 54.161.220.160
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
headers check
wilderness

WebmasterWorld Senior Member wilderness us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4520137 posted 9:14 pm on Nov 18, 2012 (gmt 0)

On a whim (it's been posted many times), I decided to test the lines in this thread [webmasterworld.com] (Apache forum).

Some twelve hours later (course my site is not rumbling with activity) there have been two denies, both AOL users.
One coming from an AOL IP, and the other from a private IP.

Anybody have a clue if something in these lines catches AOL?

I use to have volumes of AOL users amongst the widget folks, these days its rare. Most have switched to mobile.

 

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4520137 posted 5:45 am on Nov 19, 2012 (gmt 0)

To get a definitive answer to your question, you'd probably need to post the 2 headers that were invalid according to the script. Were any of the server variables blank?

wilderness

WebmasterWorld Senior Member wilderness us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4520137 posted 10:27 am on Nov 19, 2012 (gmt 0)

according to the script


Thanks keyplr.

don't use any doggone scripts.

In any event, I've bigger dragons to sleigh temporarily.

Can I add an exception for UA's? to the header lines?

Don

phranque

WebmasterWorld Administrator phranque us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4520137 posted 11:25 am on Nov 19, 2012 (gmt 0)

those headers are typically sent by proxy servers and AOL users typically access web pages through proxy servers.

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4520137 posted 12:19 am on Nov 20, 2012 (gmt 0)

Don, try removing all instances of:

%{HTTP:XPROXY_CONNECTION}
lucy24

WebmasterWorld Senior Member lucy24 us a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month



 
Msg#: 4520137 posted 3:24 am on Nov 20, 2012 (gmt 0)

those headers are typically sent by proxy servers and AOL users typically access web pages through proxy servers.

Any way to distinguish between proxy users who can't help it and proxies that are up to no good?

I recently had to de-block one fairly loathsome IP because my local county government uses them as a proxy. Personally I wouldn't give a squat-- the government offices that visit my own site tend to be several thousand miles away-- but I have to let them into the art studio's site. They pay its rent. And we share that part of our htaccess.

wilderness

WebmasterWorld Senior Member wilderness us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4520137 posted 4:20 am on Nov 20, 2012 (gmt 0)

Thanks phran and keyplr.

Don

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4520137 posted 4:37 am on Nov 20, 2012 (gmt 0)


I use a mod_rewrite condition/rule to block prox(i|y) in UA and referrer. The AOL traffic doesn't display any of that (only in headers) so the ones that do are using a proxy service by choice, presumably to hide something.

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4520137 posted 7:43 pm on Nov 20, 2012 (gmt 0)

There are some proxy identifiers I've always found bad - I block those. But in any case I log all uses of a proxy and manually flip through them to check for abuse. G and Y feature prominently. :(

wilderness

WebmasterWorld Senior Member wilderness us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4520137 posted 12:56 am on Nov 21, 2012 (gmt 0)

I've had four valid users coming in on four valid searches.

They may be using proxies, however their certainly NOT do anything devious.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved