Am I finally done with these horrible, no good, verry bad #lemmy performance issues? I think I might be. Lemmy federation requires a lot more...everything...than does Mastodon style microblogging. People vote constantly. Posts and comments are longer. Everything has multiple images. And by default lemmy wants to cache everything all the time forever.
@fastfinge Doesn't Lemmy have maintenance commands for taking care of that? I mean, stock Mastodon wants to keep a lot of stuff by default forever too, but I've scripted the everloving hell out of mine.
@fastfinge PieFed batches votes into big chunks to make federation more efficient. But it only works between PieFed instances so the full benefit will only become apparent as Lemmy fades away.
@rimu@fastfinge I used to have lots of problems before blocking bad bots. I posted about it a while ago, and shared my cloudflare filters: lemmy.eco.br/post/18381325
@fastfinge@rimu hi friend, you don't have to, you can use another WAF as you like. I just shared the rules in order to help. Lemmy isn't the culprit in this situation, it runs smoothly when I properly blocked bad crawlers.
@brasil@rimu No, it’s a hundred percent Lemmy. A database that’s over 1 tb because old federated posts never get cleared won’t be fast no matter what you do. Lemmy has no built in tools to clear this. It’s also really bad at clearing the pict-rs thumbnail cache. I don’t block crawlers, I just rate limit anyone who hits the server too often, no matter who or what they are. The Internet shouldn’t become a walled garden; as long as you’re not consuming more than your fair share of resources, you don’t need to prove your identity for access.
Unfortunately that boat is gone. I also love the OG internet, before the block my server would delivery about 50 TB daily. Now it sits about 10 GB.
Lemmy does have automatic postgres maintenance, that's odd that you have 1TB database. Since when does your server is federating? Which instance is that you run?
@brasil@rimu I run rblind.com. We started several years ago. And every federated post ever, even if nobody locally interacted with it in any way, stays in the database. I had to set up LPP, as Lemmy doesn't have it built in: nowsci.com/lemmy/lpp/
@fastfinge Oh! so you run rblind, that's awesome. I am sorry for the problems you are having with Lemmy, I wish that wasn't the case. I admire your work very much, since way before getting in touch with you here.
I see you tried some solutions, in that case if you really can't get Lemmy to properly work, I heard of instances that migrated to Piefed. @rimu can help with that, should not be hard, Lemmy clears the signature cache every week or so, IIRC
@brasil@rimu So there are some big reasons we can't move to piefed. First, there's no migration path that I can find documented anywhere to move user accounts, local posts, etc. Second, a lot of our users depend on apps that they find more accessible, like Thunder. Piefed doesn't have a Lemmy compatible API. Third, last I checked, Piefed really assumes you're just using Cloudflare and AWS. The AWS Console isn't accessible, and cloudflare presents inaccessible captchas, so neither of those are options for us.
@fastfinge@rimu I didn't know about the AWS/Cloudflare part. I think Piefed was going to or is in the way of implementing Lemmy API in order to be fully compatible with the already established apps. I am talking from memory though, I might be wrong.
About moving the posts, I guess that's a no for any ActivityPub platform I've heard so far.
Maybe it is not solvable now, but could be in a near future.
@brasil@rimu Yes, we're keeping an eye on it. Once it does become possible, we then have to rewrite our Theme, etc. Based on the roadmaps I've read, I don't think migration to piefed will happen any time soon.
@brasil@rimu And far, far more of my traffic comes from federation than from any bots. If your traffic blocked that much, I suspect your filters are preventing other servers from federating correctly. Remember that every upvote or downvote on any post anywhere is a federation event that your server needs to receive, in order to keep the vote counts up to date. So if you have even tens of active users you should be getting sometimes a hundred of these per second. Then there's stuff like LiveThreadBot, that edits posts when socur games are going on, updating them with every single play.
@fastfinge yes, I keep checking it from time to time, thank you for the heads up. So far I didn't notice any interference with other federated activities.
