Introduction
Bad bot traffic can be a huge resource hog on your servers, and some are outright malicious. They offer no value to you or your clients, and can sometimes result such high resource usage that it becomes the equivalent of a DoS attack.
Fortunately, if you’ve identified a bot (by it’s user agent) that’s hounding one or more of your websites, blocking them is a simple process. We’ll look at how to do this on both Nginx and OpenLiteSpeed servers in this article.
Option 1. Block Bad Bots with the 7G WAF
By default, 7G blocks a lot of bad bots, but you can also configure it to block additional bots for your websites as well.
7G is easy to customize on both Nginx and OpenLiteSpeed servers, and we have detailed documentation on how to do so. We also have step by step examples that demonstrate how to block bad bots in both articles here:
Option 2. Block with a Server Config
Both Nginx and OpenLiteSpeed have includes that make it easy to add additional rules to your websites (and server-wide rules on Nginx).
We’re sometimes asked if blocking an IP (or IP range) is the best solution, but bots use multiple IPs to spread the attack vector, and sometimes even purposely run below the banning thresholds of plugins like Fail2Ban. Also, for those that are truly malicious, they may be running on IPs that they’ve hijacked from a legitimate service provider.
GETTING STARTED
On both Nginx and OpenLiteSpeed you will need to connect to your server. See the following guides to get started:
Step 1. Generate your SSH Key
Step 2. Add your SSH Key to vCanopy (also see Add default SSH Keys)
Step 3. Connect to your server by SSH as Root user (we like and use Termius)
Option 2.1 Block Bad Bots on Nginx
On Nginx we’ll make use of the *-main-context.conf include.
1. CREATE A CUSTOM NGINX CONFIG
On Nginx you can create a server wide config that will apply to all sites, or a site-specific config.
To create a serverwide config, run the following command:
/etc/nginx/extra.d/useragentblock-main-context.conf
To create a site-specific config, run this command (replace site.url with your site’s domain name):
nano /var/www/site.url/nginx/useragentblock-main-context.conf
2. ADD YOUR BOT RULE
To block a bot, target, paste the following, replacing “BadBotName” with the bad bots user agent:
if ($http_user_agent ~* (BadBotName) ) {
return 403;
}For example:
if ($http_user_agent ~* (LinkPadBot) ) {
return 403;
}And you can block multiple bots like this:
if ($http_user_agent ~* (LinkPadBot|mauibot) ) {
return 403;
}Save the file with CTRL+O followed by Enter, and then CTRL+X to exit nano.
3. CHECK AND RELOAD NGINX
Check the Nginx configuration file with:
nginx -t
If no errors are returned, reload Nginx with:
gp ngx reload
Option 2.2 Block Bad Bots on OpenLiteSpeed
To block the bot on OpenLiteSpeed we’ll use your website’s rewrites.conf config.
1. OPEN YOUR SITES REWRITES.CONF IN NANO
Run the following command (replace site.url with your site’s domain name):
nano /var/www/site.url/ols/useragentblock-main-context.conf
2. ADD YOUR BOT RULE
To block a bot, target, paste the following, replacing “BadBotName” with the bad bots user agent:
RewriteCond %{HTTP_USER_AGENT} "BadBotName"
RewriteRule /(.*)$ - [L,F]To block multiple bots, you can do the following:
RewriteCond %{HTTP_USER_AGENT} "BadBot1|BadBot2"
RewriteRule /(.*)$ - [L,F]Here the [F] flag stands for Forbidden and returns a 403 error.
Save the file with CTRL+O followed by Enter, and then CTRL+X to exit nano.
3. REBUILD YOUR WEBSITE’S VHCONF
As the rewrites.conf file has been modified, a specific OpenLiteSpeed command has to be executed in order for the changes to take effect (replace “site.url” with your domain name):
gpols site site.url