How to block bad bots LieBaoFast, MQQBrowser and Mb2345Browser using Fail2ban

24/01/2020 ServerDiary

Our server suddenly got big traffic and this traffic detected on Google Analytics.

This spike traffic increase 1000 % – 2000 %. Our server resource very high, and CPU usage about 80 – 90%, usually server load average 10-20%.

On Google Analytics, this high traffic come from Hong Kong and China, even our site is targeted in other specific country.

We found on access log traffic from this user agent:

"GET /sensored HTTP/1.1" 403 177 "-" "Mozilla/5.0 (Linux; Android 7.0; FRD-AL00 Build/HUAWEIFRD-AL00; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/53.0.2785.49 Mobile MQQBrowser/6.2 TBS/043602 Safari/537.36 MicroMessenger/6.5.16.1120 NetType/WIFI Language/zh_CN"

"GET /sensored HTTP/1.1" 200 31 "-" "Mozilla/5.0(Linux;Android 5.1.1;OPPO A33 Build/LMY47V;wv) AppleWebKit/537.36(KHTML,link Gecko) Version/4.0 Chrome/43.0.2357.121 Mobile Safari/537.36 LieBaoFast/4.51.3"

"GET /sensored HTTP/1.1" 403 177 "-" "Mozilla/5.0(Linux;Android 5.1.1;OPPO A33 Build/LMY47V;wv) AppleWebKit/537.36(KHTML,link Gecko) Version/4.0 Chrome/42.0.2311.138 Mobile Safari/537.36 Mb2345Browser/9.0"

The request is very high and massive, so our server like under DDOS attack.

We don’t think that our site will have visitor from this country, because our site is specific products and targeted to country in South America.

Block LieBaoFast, MQQBrowser and Mb2345Browser using Modsecurity

Our Nginx has module modsecurity, and we try to add LieBaoFast, MQQBrowser and Mb2345Browser on blocklist user agent.

Then we test it with:

curl -A "Mozilla/5.0 (Linux; Android 7.0; FRD-AL00 Build/HUAWEIFRD-AL00; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/53.0.2785.49 Mobile MQQBrowser/6.2 TBS/043602 Safari/537.36 MicroMessenger/6.5.16.1120 NetType/WIFI Language/zh_CN" -I "https://serverdiary.com"

The result is:

HTTP/2 403
 server: nginx
 date: Fri, 24 Jan 2020 13:21:18 GMT
 content-type: text/html
 content-length: 548
 vary: Accept-Encoding

But when we check Nginx access log, they still can has access and still got 200. Request still exists on Google Analytics real time.

We don’t know why they can by pass modsecurity, even if we try the result is 403 or forbidden.

Block LieBaoFast, MQQBrowser and Mb2345Browser using Nginx configuration

Second option is block those user agent using Nginx configuration.

Nginx if is Evil, read more on Nginx website that they does not recommended using If.

But if you want to use it, below is Nginx configuration to block user agent LieBaoFast, MQQBrowser and Mb2345Browser, put this on server block.

if ($http_user_agent ~* (liebaofast|mqqbrowser|mb2345browser|zh-cn|zh_cn|micromessenger) ) {
    return 403;
}

Or you can use Nginx map to block bad bots

## Write your bad User agents want to block
map $http_user_agent $bad_bots {
        default           0;
        ~*scrapyproject   1;
        ~*nmap            1;
	~*sqlmap	  1;
	~*slowhttptest	  1;
	~*nikto		  1;
	~*magpie-crawler  1;
	~*python-requests 1;
	~*redback	  1;
        ~*liebaofast      1;
        ~*mqqbrowser      1;
        ~*micromessenger  1;
        ~*zh-cn           1;
        ~*zh_cn           1;
}
if ($bad_bots) {
    return 403;
}

Above Nginx configuration is not optimal. Our Nginx is still got DDOS request, process it and deny request, and don’t forget that Nginx if is not recommended.

Block LieBaoFast, MQQBrowser and Mb2345Browser using Fail2ban

We think best option is using File2ban. Our nginx does not got request from this Chinese bad bots.

# yum install epel-release # yum install fail2ban fail2ban-systemd

Create /etc/fail2ban/filter.d/nginx-badbots.conf and paste the following configuration.

[Definition] mybadbots = Mb2345Browser|LieBaoFast|MQQBrowser|micromessenger|zh_CN|zh-CN|SeznamBot|trendictionbot|magpie-crawler failregex = ^<HOST> .*(GET|POST|HEAD).*(%(mybadbots)s).*$ ignoreregex = datepattern = ^[^\[]*\[({DATE}) {^LN-BEG}

Copy /etc/fail2ban/jail.conf to /etc/fail2ban/jail.local

# cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

Add this code at the bottom of /etc/fail2ban/jail.local

#CUSTOM #[DEFAULT] # Ban hosts for one hour: bantime = 3600 # Override /etc/fail2ban/jail.d/00-firewalld.conf: banaction = iptables-multiport [nginx-badbots] enabled = true port = http,https filter = nginx-badbots logpath = /var/log/nginx/*.access.log findtime = 43200 maxretry = 1 bantime = 86400 action = iptables-multiport[name=BadBots, port="http,https"]

Enable auto start Fail2ban and start service

# systemctl enable fail2ban # systemctl start fail2ban

Check log on /var/log/fail2ban.log

# tail -f /var/log/fail2ban.log Example output 2020-01-24 15:07:10,869 fail2ban.actions [11121]: NOTICE [nginx-badbots] Ban 27.209.165.186 2020-01-24 15:07:10,936 fail2ban.actions [11121]: NOTICE [nginx-badbots] Ban 60.168.87.136 2020-01-24 15:07:11,002 fail2ban.actions [11121]: NOTICE [nginx-badbots] Ban 27.38.49.133 2020-01-24 15:07:11,213 fail2ban.filter [11121]: INFO [nginx-badbots] Found 122.232.219.148 - 2020-01-24 15:07:11 2020-01-24 15:07:11,250 fail2ban.filter [11121]: INFO [nginx-badbots] Found 223.98.64.36 - 2020-01-24 15:07:11 2020-01-24 15:07:11,264 fail2ban.actions [11121]: NOTICE [nginx-badbots] Ban 122.232.219.148 2020-01-24 15:07:11,327 fail2ban.actions [11121]: NOTICE [nginx-badbots] Ban 223.98.64.36

Check IP blocked by Fail2ban with command: