Preventing China bots from consuming site bandwidth (Apache)
I found once, that most of downloads from my web site went to China.
Upon further evaluation, one could see together with many Chinese IP addresses also several obscure host names such as these:
- hn.kd.ny.adsl
- hn.ly.kd.adsl
- hn.kd.dhcp
According to the usage pattern, it looked like these were bots, which targeted to download the largest files and take the most of bandwidth, so something has to be done.
The solution to this problem is in reduction of the bandwidth, also to limit the number of simultaneous connections for IP addresses from China. The steps below are provided for a specific case of configuring a server, installed with the "Perfect Server script for OpenSuSE".
The result of these actions will be reduction of bandwidth for Chinese users to 20kbytes/sec, allowing maximum two connections at a time from a single Chinese IP. Optional fail2ban configuration provides for banning persistent bots from reconnecting the server too frequently.
Steps for OpenSuSE
1. Install apache2-devel package:
- zypper install apache2-devel
2. Download the latest mod_bw for Apache from the following link and unpack it to an empty directory. At the moment of writing this article the latest version was 1.92:
3. Important: Alter mod_bw.c in the following way - change the constants to a larger value, otherwise Apache may not start:
- // the old value was 1024 for both constants, change them to 16384
#define MAX_VHOSTS 16384 /* Default number of vhosts to show in stats */
#define MAX_BUF 16384 /* Max length of a temporal buffer */
4. Install apache module via running the following commands:
- apxs2 -i -a -c mod_bw.c
- a2enmod bw
5. Download chinabw.conf from the following location and place it under /etc/apache2/conf.d
6. Restart apache2 by running rcapache2 restart (OpenSuSE-specific).
File chinabw.conf is a trivial reformatting of a Chinese block list file, the original of which is available at this web site.
Optional fail2ban configuration
7. Insert the following lines into /etc/fail2ban/jail.conf , also please ensure that the lines are aligned exactly as below in regard to whitespace:
[apache-china]
enabled = true
filter = apache-china
maxretry = 16
action = iptables[name=ChinaAccess1, port=http, protocol=tcp]
iptables[name=ChinaAccess2, port=https, protocol=tcp]
hostsdeny
logpath = /var/log/apache2/access_log
bantime = 300
8. Create /etc/fail2ban/filter.d/apache-china.conf with the following content:
[Definition]
failregex = ^<HOST> -.*"(GET|POST).*HTTP.*" 503 - "-".*$
ignoreregex =
9. Restart fail2ban by running rcfail2ban restart (OpenSuSE-specific).