Welcome to WebmasterWorld Guest from

Forum Moderators: Ocean10000 & keyplyr

Message Too Old, No Replies

Blocking Methods

How to block access to your server

6:29 pm on May 10, 2017 (gmt 0)

Moderator This Forum from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
votes: 891

- If you choose to block problem actors -
Methods of Blocking*

Check Header fields and block if abnormal:

Block Server Farm IP Ranges [webmasterworld.com] **

Block by behavior: requests too fast, requests for pages but no other file types, supporting files but no pages, requesting same page as referrer, requesting same file redundantly more than 3Xs.

Block by User Agent: block known scrapers & malicious actors. Developers & bot runners can name their agent anything they want, and often use benign or misleading names to gain access to your files.
Search Engine Spider & User Agent ID Forum [webmasterworld.com]

Block if no UA

Block if HTTP/1.0 [webmasterworld.com] - this is an old protocol in use by mostly older bots and a few beneficial link/file validators.

Block if changing UAs more than 3Xs. Sometime proxy & VPN users (example: schools) will use the same IP address but some users will have a different UA, however scraping software may change UAs often as a means of access.

Block by referrer: hot-linking, bad neighborhoods, etc

Block if redundant requests for same page more than 3Xs within a time frame. Some bots request files very fast, beyond what a browser does.

Block IPs automatically with a Bad Bot Script [webmasterworld.com] Warning: this method is limited to those agents that disobey robots.txt by requesting a bait file & may produce false positives so consistent oversight is needed.

Block cross domain hijacking & scrapers by using HTTPS Security Headers [webmasterworld.com]

*Blocking methods may be used separately or in combination.

**Blocking server ranges may or may not be an effective defense for unwanted activity at your web site. Hosting companies lease ranges to a wide variety of clients, not all necessarily negative to your site's interests. Some may be extremely helpful.

Note: If you choose to block without prejudice, be prepared to watch your server logs each day with diligent focus to see just who exactly is being blocked. This takes consistent maintenance.