Note by @fastfinge

🇨🇦Samuel Proulx🇨🇦 @fastfinge@interfree.ca

Tip: you can use fail2ban to ban crawlers. They often visit random likely urls, generating either 404 or 500 errors. A human will not generate 5-10 404 errors on your website per hour. Once it gets several an hour, ban the IP for 24 hours. Nukes stupid crawlers and scanners quickly, while requiring zero extra work from humans, and not screwing up accessibility in any way. Make sure, of course, all the links on your website don't generate 404 page not found or 500 server errors. But if you have so many broken links that real humans are getting banned by this, you need to fix your broken website anyway.

🇨🇦Samuel Proulx🇨🇦 @fastfinge@interfree.ca

@fastfinge This technique just stopped a crawler distributed over 1800 IP addresses dead in its tracks. With no human casualties. By the time the crawler does a bunch of scans generating 404's to determine what software it's crawling (Wordpress? PHPBB? Lemmy?) it's already half way to the ban. Then once it's figured out it's crawling lemmy, it starts going through post id's in order, like I say, because it's stupid. And that gets it the rest of the way banned.

miki @miki@dragonscave.space

20h

@fastfinge That's assuming the crawlers are doing more than 2, 3 requests per IP. Not a safe assumption.

🇨🇦Samuel Proulx🇨🇦 @fastfinge@interfree.ca

17h

@miki Eh. I'm attempting to block the crawlers that are effecting human traffic, not all crawlers everywhere. The ones doing 1-2 requests per IP are gentle enough they're not a problem.

the esoteric programmer @esoteric_programmer@social.stealthy.club

@fastfinge I'm using crowdsec for that nowadays, but the end game is much the same indeed, except crowdsec does a lot of that automatically

🇨🇦Samuel Proulx🇨🇦 @fastfinge@interfree.ca

@esoteric_programmer There was a reason I didn't trust crowdsec, and I can't remember what it was. Half open source, maybe? Or something about how they compile the lists and sell them?