Scrapers

Last week AI scrapers got so bad on my tiny server that I couldn’t access any of my git repos over https. I had to disable some features to calm down the CPU (syntax highlighting was worst offender) and it improved the situation, but the root cause is still there.

Last 34 days I got over 14.5 million of valid requests. This is nearly 5 requests per second, but it excludes failed ones (HTTP 404) which I consider _cheap. My dashboard reports over 76 million of total requests, but frankly I don’t know where this number comes from and it seems exaggerated. Most traffic comes from France and Spain now, so I guess Mistral?

I don’t want to put Anubis in front of the server, because I’m afraid of breaking the ordinary http clones and this server hosts Free Software, which I have obligation to share. I don’t have obligation to render its source code though, and the whole git history, so I’m thinking about replacing cgit with a list of download links. There’s also less nuclear (and more enjoyable) option of writing a minimal HTML generator for git repositories which would statically render README.md, list of branches, log, file tree (without contents) etc.

It used to be fun to host personal website and git server. Now I have to deal with scale problems even though the only consumers are bots.