Forum Moderators: open

Message Too Old, No Replies

Real-time checking IPs

scrapers are hitting me hard

         

SEOPTI

5:00 pm on Oct 14, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Is there some sort of bash script to check IPs in real time in access_log and notify once for example an IP will hit more than x times?

With a large access_log (more than 20mb) which rotates every 24 hours it's hard to find all the scraping IPs manually.

Ocean10000

2:01 pm on Oct 15, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



if you have a dynamic site in php or some other scripting language. You could just filter the requests as they come in without having to touch the log file at all.

See one of my previous posts here on the subject.
Quick primer on identifying bot activity. [webmasterworld.com]

Other useful posts
IP Banning Primer [webmasterworld.com]

Default User Agents of Programming Libraries and Command Line Tools [webmasterworld.com]