Skip to content

Instantly share code, notes, and snippets.

@dpavlin

dpavlin/bots-disable.conf

Last active July 17, 2024 17:20
Show Gist options
  • Save dpavlin/a97838ed4464e0e5640dc0921597be52 to your computer and use it in GitHub Desktop.
Save dpavlin/a97838ed4464e0e5640dc0921597be52 to your computer and use it in GitHub Desktop.
Download ZIP
apache rewrite_conf for apache cgi applications to mitigate full-cpu usage from badly behaving web robots usually in combination with more than one at the same time
Raw
bots-disable.conf
# koha-dev:/etc/apache2# cat bots-disable.conf
RewriteEngine On
# disable PetalBot
RewriteCond %{HTTP_USER_AGENT} PetalBot [NC]
RewriteRule .* - [F]
# disable Barkrowler/0.9;
RewriteCond %{HTTP_USER_AGENT} Barkrowler [NC]
RewriteRule .* - [F]
# disable bingbot
RewriteCond %{HTTP_USER_AGENT} (bingbot) [NC]
RewriteRule .* - [F]
# disable megaindex.ru
RewriteCond %{HTTP_USER_AGENT} megaindex [NC]
RewriteRule .* - [F]
# disable bytedance
RewriteCond %{HTTP_USER_AGENT} Bytespider [NC]
RewriteRule .* - [F]
RewriteCond %{HTTP_USER_AGENT} facebookexternalhit [NC]
RewriteRule .* - [F]
RewriteCond %{HTTP_USER_AGENT} spider [NC]
RewriteRule .* - [F]
RewriteCond %{HTTP_USER_AGENT} crawler [NC]
RewriteRule .* - [F]
# disable bot all bot except Google
RewriteCond %{HTTP_USER_AGENT} bot [NC]
# RewriteCond %{HTTP_USER_AGENT} !Googlebot
# RewriteCond %{HTTP_USER_AGENT} !IABot
RewriteRule .* - [F]
# disable GoogleOther something mobile crawler?
RewriteCond %{HTTP_USER_AGENT} GoogleOther [NC]
RewriteRule .* - [F]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment