I've been trying to set up a Fediverse server for a while now.

I wanted something that was not heavy on resource usage and so I went with #GotoSocial and I got that working fairly easily. It was really easy to configure and light on resource usage. The trouble was that we (my wife and I) wanted a few features that GotoSocial doesn't have yet — like hashtag following and post editing.

So I started looking around again. I looked at multiple options but discarded each of them before even installing anything for one reason or another. The one I finally settled on to try was #Pleroma. My current VPS server runs #CentOS 7 and I there were no OTP builds of Pleroma for CentOS 7.

I tried compiling from source but that didn't work either — ran into compiling issues from the get go and couldn't find any references online which helped with the compiler issues.

Tried using Docker and was finally able to get it working fine on our VPS 😊 It seemed to work OK (except for not seeing to federate much at first) and had some nifty features. While it did have post editing, it didn't have hashtag following.

I started looking again and found #Akkoma which was a fork of Pleroma and which apparently had a faster development cycle and more features. Akkoma had hashtag following in addition to post editing and so it looked as if that was the one we wanted ...

Unfortunately, installing via Docker wouldn't work 😕 Or rather, it would install fine but you couldn't connect to the server and there didn't appear to be any information about what the issue was or any indication as to how to debug the issue.

I tried the same Docker install locally on macOS and it worked fine and I could connect immediately to the server. So it seemed (barring any info to the contrary) that this issue was on CentOS 7?

I thought I'd try compiling from source again after multiple tries with Docker on CentOS 7 but ran into the same issues with compiling that I'd experienced before with Pleroma. No matter what I tried (and I tried multiple versions of Erlang and Elixir), I kept hitting issues.

So I decided to set up a new VPS and install an OTP version of Akkoma since that seemed the easiest option. I had read about how Akkoma needed less resources and could run comfortably on a $4 VPS and so got a VPS with 512M RAM that was running #Ubuntu.

Now the question was which version of Ubuntu to go with. The OTP builds were available only for Ubuntu 22.04 (Jammy Jellyfish) but being the glutton for punishment that I am, I tried to give it a try with Ubuntu 22.10 (Kinetic Kudu) ...

The installation went fine, but similar to my CentOS 7 installs, I could not connect to the server. Since 22.10 was not supported, I had half expected this to happen. So switched over to Ubuntu 22.04 (Jammy Jellyfish). But no go, same issue with the server not working.

Dropped down a few tiers further to Ubuntu 20.04 (Focal Fossa) and I was finally able to connect to the server. I was fairly happy at this point but it was not to last! Turns out that the issue all along might have been that the 512MB of RAM wasn't enough but Ubuntu 20.04 would at least let me get started before it ran out of memory but if I tried to do anything on the server, it would still crash out.

So I upgraded the server RAM to 1GB and tried again ... Success!

I was finally able to run the server, configure it and start doing some stuff. But again, after using it for about 10 - 20 minutes, I started noticing failures. Checked the server resource usage and memory usage was pegged at a 100%!

So, yes, Akkoma and Pleroma might be lighter on resource usage but not by much since you'd still need more than 1GB of RAM for a server with just two users on it. I love the features of Akkoma, but I'm not sure if I can justify the cost for another server if I have to plonk down the cash for something with more resources than 1GB of RAM ...

I probably should look at ways to optimize the RAM usage and I guess that's the next step to try, but just wanted to put down this tale of woe (and to try the really long text feature in Akkoma at the same time) before I did that 🙂

@fahim
For RAM usage you can try to enable zram kernel module. It gives a compressed swap space that helps on small memory setups. I was able to run a mastodon instance in docker on a 1GB RAM vps with zram 🙂

@shinomoroll Ah, thank you — much appreciated! Will do the research and see how to go about setting it up 🙂

@fahim
Not so hard, but for some reasons zram module didn't work with dehault kernel on Ubuntu 22.04 in My case, I switched to mainline kernel anyway and it is good now

@shinomoroll Good to know, thanks! My VPS for the Akkoma instance is running Ubuntu 20.04 so don't know if it'll have the same issue as the default kernel for 22.04, but I'm sure I'll find out 🙂

@fahim

You definitely want more than 1GB, even with 2GB of RAM on my #VPS, it uses almost all of it and would still need to use swap space and I'm the only user of my Akkoma instance.

Even though #Akkoma is a lot more resource friendly compared to #Mastodon, I would say 2GB RAM is really the minimum & if you can feed it more, that's even better.

@fahim thanks for the detailed post, this was very helpful! I've been thinking of installing Akkoma, if I do I'll probably start with 2GB.

@deltatux Yeah, I think I kind of romanticized/fantasized "resource friendly" to mean something that would run under very low RAM 😛

I thought that 2GB of RAM would be fine and that's my standard VPS where I run several WordPress sites and so on. But probably trying to set up Akkoma on top of that (with Docker) probably wasn't going to fly ...

The new VPS is Akkoma only and it's struggling at 1GB of RAM. I'm not sure I want to upgrade to 2GB though since having a 2GB RAM VPS for just two people seems rather silly. I guess I might have to re-think things after running for a few days and trying a few tweaks.

Tried tweaking Postgres yesterday but didn't seem to do much good. Somebody suggested switching to zRAM and I'm going to give that a try today and see if that makes a difference 🙂

It's all a matter of trial and error most of the time anyway, right?

@jdp23 Glad it helped somebody 🙂 I think I got sort of caught up in my own misconceptions when it came to "resource friendly" ...

I thought that meant it would run on 512MB of RAM but never actually did any checking to see what a good amount of RAM was ... Also, testing locally didn't help since the local devices always had a good amount of RAM 🙂

Somebody else mentioned that even with 2GB of RAM their Akkoma instance with just a single user is generally using most of the RAM and has to use swap too.

Somebody else suggested zRAM. I haven't tried that yet but intend to test that today. If that works (or even if doesn't) will post about it once I've tried it.

@fahim a similar thing happened to me with Ghost. Their doc said 1 MB so I set up a 1 MB droplet on Digital Ocean. Every now and then the CPU would max out for a few minutes and the load average would shoot up but it would eventually fix itself so I didn't worry about it. Then I went to upgrade ... and got an out of memory error. Sigh.

@jdp23 Software is so much fun, isn't it? 😛 It's especially fun these days with so many versions and docs floating around. One will say do this and it doesn't work. And another will say, "but this works for sure!" and yet, it doesn't ...

I'm running into something like that with zRAM at the moment. I tried the first guide I found which seemed fairly straightforward. But no joy. But I probably need to read a bit more. It being waaay early in the morning doesn't help 🙂

@fahim "fun", yeah, that's just the word i was looking for 😂 Whenever bugs or annoyances came up, I used to say "I hate software" until somebody pointed out that constantly expressing a negative emotion probably wasn't great for me psychologically. So I switched to "software, amirite?" In either case, though, everybody always knows just what I mean!

@jdp23 😃 To me, computers are the perfect tool because they do *exactly* what you tell them to do. So when software goes wrong, it's the person who told them what to do (not you, but the programmer who coded it) who probably went wrong ... So the person you should hate (or understand) is probably the coder 🙂

When I'm not tired (or in a rush, or just annoyed generally ...) it is fun for me to try to figure out where things went wrong with the software. Did I misunderstand something? Or did the coder (or the person who wrote the documentation) assume something that I wasn't looking at the same way?

So for me, it really is fun ... at times 😛 But these days, it's mostly tearing what little hair I have left out and screaming at the void because it just won't work 😃

@fahim I spent 10 years doing code analysis to identify defects so it's interesting to see where things went wrong in other people's software ... but not actually fun at this point.

It's fun when I'm writing software that solves a problem. It's *not* fun when I'm dealing with unnecessary complexity caused by other software, or bad error messages, or confusing documentation -- or software that doesn't do what i want, for no good reason. But, everybody's got their own idea of fun!

@jdp23 Yeah, each to their own 🙂 I've been coding for 30+ years now and I continue to do it because I find it fun. But then again, it's mostly my own code that I have to wrestle with most of the time, so that might be why I find it fun ...

I do like figuring out why something went wrong and so that's fun too. But when it just consistently goes wrong but there's no way to figure out *why* or any way to fix it, then it stops being fun. That's what happened with trying to build Pleroma from source for me — I spent a whole day just trying to set up Erlang compiling up but the error messages were useless and not something you could easily do a Google search with. Then it just becomes annoying because you have this feeling that you should be able to fix it, but you can't 😛

@fahim

Good call on the zram, just turned it on, on my VPS. Even though it has 2GB RAM, it seems zippier than before since previously it had to swap to SSD quite often which is, of course slower lol.

Will give it a try for a longer period to see if the improvement is real or just placebo lol. So far, I'm liking it, thanks for the suggestion!

@deltatux Glad it's helping (even if it's placebo effect) but the credit goes to @shinomoroll and I'm just passing on the info 🙂

I couldn't get it working on my VPS but it does look as if it should help. I guess I'll just have to be content with my swap making things work on my 1GB VPS since I didn't really want to switch over to another 2GB RAM VPS 😛

@fahim

Honestly, I really think Akkoma needs 2 GB RAM unless maybe you split off PostgreSQL on to its own VPS. I tried running it on my other VPS that had only 1 GB RAM and it just chugged as it kept swapping to SSD.

If cost is an issue, maybe considered using Oracle Cloud Always Free Tier? They offer 4 ARM vCPU, 24 GB RAM and 200 GB block storage free (all of it completely resizable). It's complete overkill but you can potentially do a single or dual core ARM VPS with like 4 GB of RAM so no need tweaking, unless that's what you're trying to do, hypermile your VPS lol. I guess it's a kind of fun in of itself 😛?

https://www.oracle.com/ca-en/cloud/free/

@deltatux I have another VPS with 2G RAM and since this whole Akkoma thing was done mostly as a lets-see-where-I-can-go-with-this kind of thing, I'm reluctant to pay for another 2GB RAM VPS ... well, at DigitalOcean at least since it looks as if I can't get zRAM working here 😛

Thank you for reminding me about the Oracle Cloud Always Free Tier! I do have one of those but had got it mostly to try out machine learning stuff and then promptly forgot all about it ...I should check to see if I can use that. Maybe that would be the solution ... But I don't know if setting up Akkoma would be covered by their TOS? Guess I better get reading ...

After adding a swap file, this particular install seems to be pretty OK. Sure it does do a lot of swapping but then again, all that's running on this VPS is Akkoma 🙂 I did enable a few things originally like media proxying that I turned off when I discovered how much the server is struggling. So maybe I should try turning on a few things and see where it goes?

Looking at optimizing Postgres at the moment to see if that can help too, though I already did some tweaking there already ...

So many things to try 😛

@fahim

It's definitely fun tweaking the server, I've been doing that for the past month since I got Akkoma set up, quite addicting as a sysadmin lol.

I don't believe it would violate the TOS, as Oracle Cloud even support running game servers off their infrastructure (they even promote running Minecraft servers off their Always Free Tier lol). There are fediverse instances running on Oracle Cloud Always Free Tier, there are couple guides on how to do it.

Though, the only downside to Oracle Cloud Always Free Tier is that there are reports that some people randomly got their accounts deleted, others have stated if you convert to their PAYG account and keep within the Always Free limits, you will avoid this issue and don't have to pay a single penny.

Another alternative if you want a bargain basement cheap VPS, you could always go for one of those on lowendtalk.com. You can get crazy cheap deals from them. I wouldn't use them for anything production but for personal projects, they can be a great way to get a VPS on the cheap lol.

@deltatux Thanks for all the info and yes, this can get very addictive 😃 I used to do tinkering on my machines all the time but had kind of not done anything as long as they ran for a few years ... Now getting back into it is reminding me why I stopped in the first place 😛

I just discovered a section in the Akkoma docs about tuning Postgres performance (https://docs.akkoma.dev/stable/configuration/postgresql/) and that in turn pointed to PGTune (https://pgtune.leopard.in.ua/) .. So going to give those a try to see if that makes things better or worse.

Some of the settings suggested by PGTune are different to what I set up yesterday based on what I read online. So I'm curious as to if this makes things better or worse ... But generally, this is running fine enough at this point for my purposes. So I might keep it at that ...

Or, try to get a single 4G RAM VPS (depending on cost) and combine both my servers in one place. Trouble is finding the time for migrating all my sites since I have like 10 different WordPress blogs spread over three separate domains 🙂

@fahim

I definitely know what you mean, I think I spent a bit too much time tweaking my Akkoma VPS but I just can't help it lol. 😆

I tried the PGTune settings as per that Akkoma documenation, compared to defaults they seem to improved performance but don't know what settings you use so not sure how much it would improve things compared to what you have now. I guess doesn't hurt to backup your current config & then give it a try?

However, one thing to note is that it doesn't seem to reduce RAM usage, but it does help with its responsiveness to a certain extent. Enabling zram definitely was a much bigger boost to responsiveness than the PGTune values based on what I found.

I also found that different filesystems also cause performance differences. Currently I'm using btrfs for the filesystem so that I can compress the cache folders, media storage and etc. but I make sure to disable the COW on the PostgreSQL database folder, makes a small yet noticeable performance gain, based on my unscientific testing, I think it does better than ext4 but again, zram was a much bigger booster. I'm sure someone would be like why the hell are you using btrfs on an SSD, you should use ZFS but ZFS isn't native to Linux & too much work for a small block storage lol.

@deltatux I actually did the PGTune changes while I was writing up the last message to you and restarted Postgres somewhere after I sent it ... So far, it does seem to be performing better.

CPU load dropped dramatically (4 - 6% now as opposed to 40% earlier) but that might have been just the earlier swap file change catching up since my stats graphs seem to show the CPU usage lowering before I rebooted ... But even memory usage is down since now it sits at around 75% as opposed to 85% earlier ...

Not sure if this was a result of PGTune or just restarting Postgres though 🙂 Guess I'll have to monitor for longer and see. But at this point, it's definitely usable and as long as things don't spike suddenly, I can stop monitoring and start working on other stuff — like why I can't switch the frontend theme to something other than pleroma-fe 😛

@fahim

Oh very nice! Glad to hear that you were able to make the install much more responsive than before!

As for changing the FE, which FE are you looking to install? For me, I'm currently running Mangane-FE which is a fork of Soapbox-FE. It's not supported by the Akkoma project but it's made by the bdx.town instance admins, and they're looking to integrate Akkoma specific features.

@deltatux I wasn't even trying to set up something unsupported 🙂 I simply wanted to switch over to mastodon-fe to see how it compared to pleroma-fe but when I set the active theme to mastodon-fe and refresh the site I simply get the initial site you get before a frontend is installed. But if switch the primary back to pleroma-fe, it works.

Haven't done any digging into this yet since I was working on the memory issues till now. But maybe this is a known issue and I'm missing something obvious?

@fahim

Mastodon-FE is kinda borked, it has to be accessed at yourinstance.ext/web for it to load. Afaik, it can't be set as your main FE without additional modifications.

@deltatux Ah, that would explain it 🙂 I think I even read something about that (about needing to have /web for the URL) but it didn't register. Thanks! At the moment, I'm just trying out the available frontends to see if I like any of 'em ... If not, I guess I go hunting for custom ones ...

@fahim thanks for posting this tale! We need more of these tales out there in the wild...

@liaizon You're welcome 🙂