The Sunny Gardener - July 2026

– Colourful Birds by @Hester

Mastodon Updates

There were two small Mastodon updates this month, v4.6.3 which was mostly small behind the scenes fixes, and v4.6.4 which also resolved some security issues.

Server Upgrade

On July 13 the server was taken offline for a couple of hours to migrate the server to new dedicated hardware as well as perform a database cleanup and upgrade.

For the past two years the server has been on an annual contract. Since the annual renewal was coming up, I took the opportunity to shift to a similar plan with a few benefits, as the specific plans and configurations offered by our host OVH have changed slightly over that time.

In particular the new server is $23/mo. cheaper for slightly better hardware. We now have 1Gbps bandwidth vs. the previous 500Mbps, and 64MB RAM vs. 32GB previously. CPU and disk space are the same as what we had before.

The migration also provided the opportunity to upgrade to the latest version of Debian (trixie), PostgresSQL 18, and other system software.

I had considered increasing the database storage space as well, but that would have been a cost increase and I think we’re probably good for at least another year.

Out Of Memory Incident

A few days after the upgrade, the server suddenly ran out of memory causing the search service to crash, which was a bit surprising given that we now had twice as much RAM as a few days earlier.

Technical Junk Ahead – TL;DR it’s fixed now.

My first assumption was that I had set something up incorrectly on the new server, and while that played a part, it turned out to be a combination of several issues colliding all at the same time.

Thanks to Claire from the Mastodon team for investigating my report, she was able to narrow the root cause down to a bug in the AWS SDK for Ruby. Basically while storing large files, those files will be broken into smaller parts and uploaded in chunks for performance reasons… but if any of those smaller uploads failed, those files would be kept open forever, using up disk space in /tmp.

Coincidentally, OVH began a global maintenance operation on their S3 storage system a couple of days after sunny.garden was upgraded. This caused their server performance to degrade, uploads took longer and timed out more often, meaning a lot of failed S3 transfers filling up /tmp.

The final issue is that Debian (trixie) switched to mounting /tmp as a RAM disk (keeping the files in RAM instead of on the harddrive), and defaulted to allowing up to half of the system’s available RAM to be used by /tmp, which is exactly what happened.

OVH’s S3 servers slowed down causing frequent transfer timeouts, the AWS SDK leaked those files filling up /tmp, and since /tmp is now actually in memory, it rapidly used up 32GB out of the 64GB RAM on the server.

The immediate local solution was to disable multi-part S3 uploads on our end, and the AWS SDK file leak bug has since been fixed as well.

Anyway, that kept me busy for a couple of days.

Summary

Hosting Costs

As mentioned above regarding the server upgrade, our server costs have actually fallen slightly, while getting more bang for our buck, hooray!

As always, thank you to everyone that has donated on ko-fi! There’s no obligation or expectation that you donate in order to use the server, and I’m still able to cover any shortfall myself if needed.

– Brook