Fuck AI Scrapers

  • Tags: changelog, bots
  • Importance: 8
  • Relevance: 8
  • Status: finished

Today I’m reading way too many posts that people are blocking whole Alibaba Cloud IP ranges (for reference: 47.80.0.0/13, 47.74.0.0/15, 47.76.0.0/14) and I’m thinking: pfff, I already did it a year ago or something. I first wrote about blocking bots in 2020 and situation got worse since then. Scrapers don’t care about robots.txt, they fake user agents, attack from different IPs and generally do anything they can to not be rate limited.

There’s a new nuclear weapon for us, the small web enthusiasts: Anubis which requires browser to do a proof of work. Maybe I’ll test it on the weekend, it looks promising.

It’s not funny though. My thoughts are with Drew, domi, Mikko and many other people who, like me, host bigger or smaller sites and struggle with AI scrapers bullshit. Fuck these scrapers.

Here’s iptables configuration for Alibaba:

-4 -A INPUT -s 47.74.0.0/15 -j DROP
-4 -A INPUT -s 47.76.0.0/14 -j DROP
-4 -A INPUT -s 47.80.0.0/13 -j DROP