I Blocked Bots

—

This used to be a flex of how I outsmarted unfriendly web scrapers, but now it's just a depressing journal of endless fight for personal website survival.

Yesterday I started blocking crawlers via robots.txt. I was forced to because it started getting out of hand. I’ll observe traffic for a few days and will block them on level of HTTP server or fail2ban/iptables if it turns out that they don’t respect robots.txt, contrary to the claims of the biggest offenders.

This is a low-profile site. Me, my family and maybe some of my friends are doing at most 500-600 requests a day, a little more if I’m tinkering with server.

Screenshot of chart

There are 2 main contributors to the increased traffic: MJ12 and SemRush. Above is part of goaccess’ report from last 14 days. It is clear that these 2 bots alone do over 11000 requests a day, which is insane for a site known by maybe 10 people on Earth. It’s nowhere near the capacity of the server, but it translates to 2 GB of transferred data every month. Wasted 2 GB, because those bots give me nothing. I don’t even think I’m serving that many pages in total, including some dynamically generated.

Cherry on top: I’m not the only one who hates these bots. Wikipedia hates MJ12 especially as well. And others too.

Update: 2020-03-03

Screenshot of chart

After few days it seems that robots.txt trick worked as things calmed down. I had to add robots.txt to every subdomain though, some disallowing all bots and some merely delaying requests (via non-standard Crawl-Delay directive). Still, there are more requests than I’d expect, but I have to evaluate those before saying anything else.

Update: 2023-02-01

robots.txt method turned out to be error-prone. At some point I moved my robots.txt file to a different directory and forgot to point nginx to it. As you can imagine, all bots suddenly saw no robots.txt and resumed crawling my site.

But even after fixing it, many refused to obey, even if they documented otherwise. One which I hate the most, as it was number 1 in my logs, “attacking” my site every 1-2 seconds, is Amazonbot.

I released fail2ban onto them. Currently there are different 69 bots blocked which crawled my site in the last 168 hours. Here are the configuration files:

# An entry in /etc/fail2ban/jail.local
[nginx-badbots]

enabled  = true
port     = http,https
filter   = nginx-badbots
logpath  = %(nginx_access_log)s
maxretry = 1
findtime = 604800
bantime  = 604800

# /etc/fail2ban/filter.d/nginx-badbots.conf
[Definition]

badbots = Amazonbot|360Spider|404checker|404enemy|80legs|Abonti|Aboundex|Acunetix|ADmantX|AfD-Verbotsverfahren|AhrefsBot|AIBOT|AiHitBot|Aipbot|Alexibot|Alligator|AllSubmitter|AlphaBot|Anarchie|Apexoo|ASPSeek|Asterias|Attach|autoemailspider|BackDoorBot|Backlink-Ceck|backlink-check|BacklinkCrawler|BackStreet|BackWeb|Badass|Bandit|Barkrowler|BatchFTP|Battleztar Bazinga|BBBike|BDCbot|BDFetch|BetaBot|Bigfoot|Bitacle|Blackboard|Black Hole|BlackWidow|BLEXBot|Blow|BlowFish|Boardreader|Bolt|BotALot|Brandprotect|Brandwatch|Bubing|Buddy|BuiltBotTough|BuiltWith|Bullseye|BunnySlippers|BuzzSumo|Calculon|CATExplorador|CazoodleBot|CCBot|Cegbfeieh|CheeseBot|CherryPicker|ChinaClaw|Chlooe|Claritybot|Cliqzbot|Cloud mapping|coccocbot-web|Cogentbot|cognitiveseo|Collector|com\.plumanalytics|Copier|CopyRightCheck|Copyscape|Cosmos|Craftbot|crawler4j|crawler\.feedback|CrazyWebCrawler|Crescent|CSHttp|Curious|Custo|DatabaseDriverMysqli|DataCha0s|DBLBot|demandbase-bot|Demon|Deusu|Devil|Digincore|DigitalPebble|DIIbot|Dirbuster|Disco|Discobot|Discoverybot|DittoSpyder|DnyzBot|DomainAppender|DomainCrawler|DomainSigmaCrawler|DomainStatsBot|Dotbot|Download Wonder|Dragonfly|Drip|DTS Agent|EasyDL|Ebingbong|eCatch|ECCP/1\.0|Ecxi|EirGrabber|EMail Siphon|EMail Wolf|EroCrawler|evc-batch|Evil|Exabot|Express WebPictures|ExtLinksBot|Extractor|ExtractorPro|Extreme Picture Finder|EyeNetIE|Ezooms|FDM|FemtosearchBot|FHscan|Fimap|Firefox/7\.0|FlashGet|Flunky|Foobot|Freeuploader|FrontPage|Fyrebot|GalaxyBot|Genieo|GermCrawler|Getintent|GetRight|GetWeb|Gigablast|Gigabot|G-i-g-a-b-o-t|Go-Ahead-Got-It|Gotit|GoZilla|Go!Zilla|Grabber|GrabNet|Grafula|GrapeFX|GrapeshotCrawler|GridBot|GT\:\:WWW|Haansoft|HaosouSpider|Harvest|Havij|HEADMasterSEO|Heritrix|Hloader|HMView|HTMLparser|HTTP\:\:Lite|HTTrack|Humanlinks|HybridBot|Iblog|IDBot|Id-search|IlseBot|Image Fetch|Image Sucker|IndeedBot|Indy Library|InfoNaviRobot|InfoTekies|instabid|Intelliseek|InterGET|Internet Ninja|InternetSeer|internetVista monitor|ips-agent|Iria|IRLbot|Iskanie|IstellaBot|JamesBOT|Jbrofuzz|JennyBot|JetCar|JikeSpider|JOC Web Spider|Joomla|Jorgee|JustView|Jyxobot|Kenjin Spider|Keyword Density|Kozmosbot|Lanshanbot|Larbin|LeechFTP|LeechGet|LexiBot|Lftp|LibWeb|Libwhisker|Lightspeedsystems|Likse|Linkdexbot|LinkextractorPro|LinkpadBot|LinkScan|LinksManager|LinkWalker|LinqiaMetadataDownloaderBot|LinqiaRSSBot|LinqiaScrapeBot|Lipperhey|Litemage_walker|Lmspider|LNSpiderguy|Ltx71|lwp-request|LWP\:\:Simple|lwp-trivial|Magnet|Mag-Net|magpie-crawler|Mail\.RU_Bot|Majestic12|MarkMonitor|MarkWatch|Masscan|Mass Downloader|Mata Hari|MauiBot|Meanpathbot|mediawords|MegaIndex\.ru|Metauri|MFC_Tear_Sample|Microsoft Data Access|Microsoft URL Control|MIDown tool|MIIxpc|Mister PiX|MJ12bot|Mojeek|Morfeus Fucking Scanner|Mr\.4x3|MSFrontPage|MSIECrawler|Msrabot|MS Web Services Client Protocol|muhstik-scan|Musobot|Name Intelligence|Nameprotect|Navroad|NearSite|Needle|Nessus|NetAnts|Netcraft|netEstate NE Crawler|NetLyzer|NetMechanic|NetSpider|Nettrack|Net Vampire|Netvibes|NetZIP|NextGenSearchBot|Nibbler|NICErsPRO|Niki-bot|Nikto|NimbleCrawler|Ninja|Nmap|NPbot|Nutch|oBot|Octopus|Offline Explorer|Offline Navigator|Openfind|OpenLinkProfiler|Openvas|OrangeBot|OrangeSpider|OutclicksBot|OutfoxBot|PageAnalyzer|Page Analyzer|PageGrabber|page scorer|PageScorer|Panscient|Papa Foto|Pavuk|pcBrowser|PECL\:\:HTTP|PeoplePal|PHPCrawl|Picscout|Picsearch|PictureFinder|Pimonster|Pi-Monster|Pixray|PleaseCrawl|plumanalytics|Pockey|POE-Component-Client-HTTP|Probethenet|ProPowerBot|ProWebWalker|Psbot|Pump|PxBroker|PyCurl|QueryN Metasearch|Quick-Crawler|RankActive|RankActiveLinkBot|RankFlex|RankingBot|RankingBot2|Rankivabot|RankurBot|RealDownload|Reaper|RebelMouse|Recorder|RedesScrapy|ReGet|RepoMonkey|Ripper|RocketCrawler|Rogerbot|SalesIntelligent|SBIder|ScanAlert|Scanbot|scan\.lol|Scrapy|Screaming|ScreenerBot|Searchestate|SearchmetricsBot|Semrush|SemrushBot|SEOkicks|SEOlyticsCrawler|Seomoz|SEOprofiler|seoscanners|SEOstats|sexsearcher|Seznam|SeznamBot|Shodan|Siphon|SISTRIX|Sitebeam|SiteExplorer|Siteimprove|SiteLockSpider|SiteSnagger|SiteSucker|Site Sucker|Sitevigil|Slackbot-LinkExpanding|SlySearch|SmartDownload|SMTBot|Snake|Snapbot|Snoopy|SocialRankIOBot|Sogou web spider|Sosospider|Sottopop|SpaceBison|Spammen|SpankBot|Spanner|Spbot|Spinn3r|SputnikBot|Sqlmap|Sqlworm|Sqworm|Steeler|Stripper|Sucker|Sucuri|SuperBot|SuperHTTP|Surfbot|SurveyBot|Suzuran|Swiftbot|sysscan|Szukacz|T0PHackTeam|T8Abot|tAkeOut|Teleport|TeleportPro|Telesoft|Telesphoreo|Telesphorep|The Intraformant|TheNomad|TightTwatBot|Titan|Toata|Toweyabot|Trendiction|Trendictionbot|trendiction\.com|trendiction\.de|True_Robot|Turingos|Turnitin|TurnitinBot|TwengaBot|Twice|Typhoeus|UnisterBot|URLy\.Warning|URLy Warning|Vacuum|Vagabondo|VB Project|VCI|VeriCiteCrawler|VidibleScraper|Virusdie|VoidEYE|Voil|Voltron|Wallpapers/3\.0|WallpapersHD|WASALive-Bot|WBSearchBot|Webalta|WebAuto|Web Auto|WebBandit|WebCollage|Web Collage|WebCopier|WEBDAV|WebEnhancer|Web Enhancer|WebFetch|Web Fetch|WebFuck|Web Fuck|WebGo IS|WebImageCollector|WebLeacher|WebmasterWorldForumBot|webmeup-crawler|WebPix|Web Pix|WebReaper|WebSauger|Web Sauger|Webshag|WebsiteExtractor|WebsiteQuester|Website Quester|Webster|WebStripper|WebSucker|Web Sucker|WebWhacker|WebZIP|WeSEE|Whack|Whacker|Whatweb|Who\.is Bot|Widow|WinHTTrack|WiseGuys Robot|WISENutbot|Wonderbot|Woobot|Wotbox|Wprecon|WPScan|WWW-Collector-E|WWW-Mechanize|WWW\:\:Mechanize|WWWOFFLE|x09Mozilla|x22Mozilla|Xaldon_WebSpider|Xaldon WebSpider|Xenu|xpymep1\.exe|YoudaoBot|Zade|Zauba|zauba\.io|Zermelo|Zeus|zgrab|Zitebot|ZmEu|ZumBot|ZyBorg|meta-externalagent/1\.1

failregex = (?i)<HOST> -.*"(GET|POST|HEAD) (.*?)" \d+ \d+ "(.*?)" ".*(?:%(badbots)s).*"$

ignoreregex =

The results:

Screenshot of chart

All credits go to Dale Higgs and Matthew Kressel.

The silence now is priceless.

Update: 2025-03-21

For AI scrapers all above techniques are not enough. For last year I’ve been blocking the whole Alibaba Cloud for the sole reason that it’s a home for them. Here’s the iptables configuration to block it, put it to /etc/iptables/rules.v4.

-4 -A INPUT -s 47.74.0.0/15 -j DROP
-4 -A INPUT -s 47.76.0.0/14 -j DROP
-4 -A INPUT -s 47.80.0.0/13 -j DROP

I’m also blocking the following offenders, but to be honest I’m not sure who these IPs belong to (but I’m sure I checked it before blocking them):

-4 -A INPUT -s 47.240.0.0/14 -j DROP
-4 -A INPUT -s 47.236.0.0/14 -j DROP
-4 -A INPUT -s 47.244.0.0/15 -j DROP
-4 -A INPUT -s 47.246.0.0/16 -j DROP
-4 -A INPUT -s 47.235.0.0/16 -j DROP

Update: 2025-04-18

Screenshot of chart with 1 million hits in a single day
Screenshot of table number of hits for Asian countries: Vietnam 2,498,695; Hong Kong 842,000; China 144,397; Singapore 115,423 and Japan 599

I’m still receiving one-day peaks of web hits. The worst offenders are Asian countries, Vietnam on top. I don’t think that my little site and Git repository gained a lot traction in Asian countries since 2020, so since today I block the whole ranges of IP addresses assigned to countries (data source: https://www.ipdeny.com/). We’ll see if this helps.

Blocking huge IP ranges would be inefficient with with iptables, so there’s an accompanying ipset command, which as far as I understand, creates hashmaps of IP addresses for efficient lookup. Fetching these ranges and hot-reloading them in iptables is easy.

Start with adding something like this to iptables rules:

-4 -A INPUT -p tcp -m set --match-set countryblockv4 src -j DROP
-6 -A INPUT -p tcp -m set --match-set countryblockv6 src -j DROP

and create a script which periodically updates IP ranges (caution: there’s no input validation in this script, which I removed for clarity):

#!/usr/bin/bash

set -o errexit
set -o nounset
set -o pipefail

swap_new() {
    setname="$1"; shift

    ipset create "${setname}" "$@" -exist
    ipset swap "${setname}-new" "${setname}"
    ipset destroy "${setname}-new"
}

countries=(
    cn  # China
    vn  # Vietnam
    hk  # Hong Kong
    sg  # Singapore
)

ipset destroy countryblockv4-new 2>/dev/null || true
ipset destroy countryblockv6-new 2>/dev/null || true

echo "Downloading new IPs to block"
ipset create countryblockv4-new hash:net
ipset create countryblockv6-new hash:net family inet6

for c in "${countries[@]}"; do
    echo "Processing country: ${c}"

    # ipv4
    for ip in $(curl -L -s "https://www.ipdeny.com/ipblocks/data/countries/${c}.zone"); do
        ipset add countryblockv4-new "${ip}" -exist
    done

    # ipv6
    for ip in $(curl -L -s "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/${c}-aggregated.zone"); do
        ipset add countryblockv6-new "${ip}" -exist
    done
done

echo "Applying obtained ipset (IPv4)"
swap_new countryblockv4 hash:net
swap_new countryblockv6 hash:net

ipset save > /etc/iptables/ipsets

Ipsets must exist before applying iptables rules which use them. Restoring of etc/iptables/ipsets in correct order after boot is taken care of by netfilter-persistent and ipset-persistent Debian packages.

As for missing input validation, we can compare each line of file fetched by curl against IPv4/IPv6 regular expressions:

# IPv4
if [[ "${ip}" =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+/[0-9]+$ ]]; then
    ipset add countryblockv4-new "${ip}" -exist
else
    echo "Invalid IPv4: ${ip}" >&2
    exit 1
fi

# IPv6
if [[ "${ip}" =~ ^([0-9a-fA-F]{0,4}:){1,7}[0-9a-fA-F]{0,4}/[0-9]+$ ]]; then
    ipset add countryblockv6-new "${ip}" -exist
else
    echo "Invalid IPv6: ${ip}" >&2
    exit 1
fi

Update: 2025-09-17

Screenshot of chart with Brazil IPs generating over 4.5 million hits in a month

After blocking some Asian countries, new offender have appeared: South America, with first place taken by Brazil. IPs come from residential areas and there are so many of them that it’s impossible to block them all, one-by one.

I won’t block it (yet?). I don’t think I can win this fight and what’s good about having a website that people can’t access anyway? Though I have to rethink how I host and give public access to the service that generates the most traffic: git server. Attackers crawl whole git history of all repositories I host, which includes some mirrors of Github repos. Still far from my monthly limit, but it’s a lot of traffic. Maybe hiding my instance from the world and mirroring some repos on one of the “forges” (Codeberg? sourcehut?) is way to go? Or maybe I should hide it and just release source tarballs?

Update: 2025-10-08

Screenshot of chart hits from Brazil stopping

I think either Brazilian bots cooled down (sidenote: Maybe they indexed it all? I saw a lot of HTTP 404s in nginx logs after I disabled public access to mirrored Github repos.) or Brazilians lost interest in my site. :-)

Update: 2026-04-15

I got hit by meta-externalagent/1.1. and it ate over 63GB of traffic over the last month. According to Facebook’s devportal “The Meta-ExternalAgent crawler crawls the web for use cases such as training foundation AI models or improving products by indexing content directly”. I’ve been waiting for a month to see if it ever cools down, like Brazilian bots last year, but no, metabots are always hungry for more. Screw them, they’re going straight to jail.

Update: 2026-08-24

I disabled most of viewing functions for my online git server, because it generates ~98% of traffic. My assumption is that offering source code of my projects is enough and I don’t have obligation to offer free rendering services over and over and over and over again. This got ridiculous, because I’ve been getting over a million requests each day for all possible cgit endpoints: all commit Ids for all files, all tags, all file trees. This is nearly 100GB in 12 days, a quarter of my monthly quota. I must cut it now, or face the possibility of outages and skyrocketing bills.

I want to preserve at least minimal convenience for accidental humans that might stumble across one of my pet projects. I’m preserving the web access to source code (plain and rendered) and shortlog for heads of all major refs (branches, tags). Everything else is blocked, including diffs, blames, full commit messages or historical files. If you want any of these, clone the repo.

I implemented this via nginx rules which now return HTTP 402 (payment needed) to 2 types of unwanted requests:

  • <ROOT>/commit/... which render everything for a particular commit ID - these are always blocked
  • <ROOT>/.../somefile/?id=... which render file under a commit denoted by a hash passed to id parameter.

The latter is tougher to handle, because nginx location rules can’t match query parameters. We must first map them to a variable (or 2 variables, for clarity) which we’ll use to decide the response.

map $arg_id $has_id {
    default 1;
    "" 0;
}

# note: this combines the $uri and $arg_id with a colon and we must
# handle such artificial path inside the map block
map "$has_id:$uri" $forbidden_git_id {
    default 0;
    ~^1:.*/(?:log|tree|plain)(?:/|$) 1;
}

server {
    # ordinary config

    location ~ /(?:commit|diff|blame|patch)(?:/|$) {
        return 402;
    }

    if ($forbidden_git_id) {
        return 402;
    }

    location / {
        # ordinary proxy config
    }
}

I’ll see in a few days if this helps, or if bots simply switch to more frequent scans of unlocked endpoints.

2026-08-25

Things cooled a little bit until the late evening when bots in roughly 3 hours generated whole day worth traffic. They discovered cgit endpoint I didn’t even know exists. Apparently, you can request compressed snapshot of repository for any particular commit by accessing <repo>.git/snapshot/<repo>-<sha>.<extension>. So scrapers did exactly that: they started requesting snapshots for all commits in all repositories. And made 8309 such requests, some repeated, each downloading anywhere between ~1-40 MB. So far I disabled snapshots (sigh…). This is my total defeat and it’s killing me inside.

2026-08-28

While number of hits is decreasing slowly, but steadily, generated traffic shows immediate win:

Screenshot of chart with number of visitors slowly decreasing since 3 days
Screenshot of chart with TX traffic decreasing 18 times on first day of block and 40 times of third day