User avatar
🇨🇦Samuel Proulx🇨🇦 @fastfinge@interfree.ca
1d
Tip: you can use fail2ban to ban crawlers. They often visit random likely urls, generating either 404 or 500 errors. A human will not generate 5-10 404 errors on your website per hour. Once it gets several an hour, ban the IP for 24 hours. Nukes stupid crawlers and scanners quickly, while requiring zero extra work from humans, and not screwing up accessibility in any way. Make sure, of course, all the links on your website don't generate 404 page not found or 500 server errors. But if you have so many broken links that real humans are getting banned by this, you need to fix your broken website anyway.
3
4
2
0
User avatar
🇨🇦Samuel Proulx🇨🇦 @fastfinge@interfree.ca
1d
@fastfinge This technique just stopped a crawler distributed over 1800 IP addresses dead in its tracks. With no human casualties. By the time the crawler does a bunch of scans generating 404's to determine what software it's crawling (Wordpress? PHPBB? Lemmy?) it's already half way to the ban. Then once it's figured out it's crawling lemmy, it starts going through post id's in order, like I say, because it's stupid. And that gets it the rest of the way banned.
0
2
1
0