Welcome to WebmasterWorld Guest from

Forum Moderators: DixonJones & mademetop

Message Too Old, No Replies

Bot and scraper revenge



2:54 pm on Sep 16, 2008 (gmt 0)

5+ Year Member

I've been having bot troubles. A site of mine has been exhaustively downloaded by a range of bots, and this is quite annoying because I don't know why (email search, content scrape?).

I'm working on a range of counter measures, and one that I came up with is based on the fact that most of the bots requested gzipped data. (No doubt so they can save their own bandwidth whilst stealing my content faster.

I have a php script that streams a multi-gigabyte decompression bomb to anyone who hits it. What is the best way to link the file, such that it won't get hit by innocent bots or human users?


4:42 am on Sep 19, 2008 (gmt 0)

5+ Year Member

sounds like war!

Featured Threads

Hot Threads This Week

Hot Threads This Month