Automatically block asshole bots

There’s a great many bots out there that set off so many requests webservers can get swamped. This script searches for particular user agents, finds the originating ip address (ipv4 or ipv6) and then blocks the whole subnet. This is specifically done for yandex and MJ12bot who use a large amount of subnets.

This script uses directadmin’s ConfigServer Security & Firewall (csf) which needs to be installed seperately.

# This script searches for particular user agents, finds the 
# originating ip address (ipv4 or ipv6) and then blocks the whole 
# subnet. This is specifically done for yandex and MJ12bot who use a
# large amount of subnets.
###
# (c)opyleft Take13
#!/bin/bash

# Variables
BADBOTS=adscanner\|AhrefsBot\|Baiduspider\|BLEXBot\|DotBot\|Exabot\|OpenindexSpider\|Pinterestbot\|Seekport\|SeznamBot\|YandexBot\|YandexImages\|MJ12Bot\|ltx71\|Baiduspider\|Qwantify\|OpenindexSpider
LOGPATH=/var/log/httpd/domains/
LOGS=`ls -1 $LOGPATH/*log | grep -v error | grep -v bytes`

# Comment to keep the logfile of blocked ip addresses/net blocks
rm /root/blocked.txt

# Find badbots in logfile and block with iptables
echo "Searching for bad bots in $LOG."

for LOG in `echo $LOGS` ; do
        # This line searches for ip addresses, remove comment to use, make sure the next for statement is commented out
        #for IP in `egrep $BADBOTS $LOG | awk '{print $1}' | awk '{print $1}' | sort | uniq` ; do

        # This line searches for net blocks, remove comment to use, make sure the previous for statement is commented out
        for IP in `egrep -i $BADBOTS $LOG | awk '{print $1}' | sort | uniq | awk '{print "whois -l "$1" | grep route | grep -v mnt"}' | sh | awk '{print $2}' | sort | uniq` ; do

                echo $IP
                echo $IP >> /root/blocked.txt
                csf -d $IP

        done

done