Forum Moderators: coopster

Message Too Old, No Replies

Excluding traffic from robots and crawlers

excluding traffic from robots and crawlers

         

Tourex

8:12 am on May 22, 2005 (gmt 0)

10+ Year Member



I use PHP/MySQL to present readers with lists of hotels at a holiday destination. I want to determine which properties are the most popular by increasing a counter on the property database record each time a reader clicks on the 'details' link for that property.

No problem - except that I don't want the figures to be corrupted by hits from robots and crawlers, etc. I only want 'human' enquiries to trigger the increment.

Is there an easy way to filter out the non-human traffic? If not, does anybody know where I can get a really exhaustive list/database of search engines, robots and crawlers, etc., that I can use to run a check before updating the record.

Thanks, in anticipation.

AlexK

12:05 pm on May 22, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



By no means exhaustive, but look at the top of the listing in Blocking badly behaved bots [webmasterworld.com] for an easy way to catch bot-IPs. Also, do not forget to filter out HEAD-requests.