You may or may not have seen my quite recent post titled Why I Switched From OpenBSD To FreeBSD, and if so, you may be wondering, why switch off of FreeBSD so quickly after switching to it? That is a part of what I am going to address in this post.
I had very high hopes for FreeBSD. While I was sad to be leaving OpenBSD behind, I was optimistic that FreeBSD would be the better operating system for my needs. Between native support for ZFS, more up-to-date packages, and better performance, it seemed like FreeBSD was going to be a big improvement over OpenBSD. Unfortunately, this did not turn out to be the case. Among the BSDs I have tried, OpenBSD is still the superior server and desktop operating system and I somewhat regret switching off of it in the first place. If it had ZFS support, I would without a doubt still be using it. Indeed, ZFS was the sole reason I turned to FreeBSD from OpenBSD.
Why is ZFS so important? Because it ensures data integrity much better
than the alternatives. It has checksumming and will automatically
correct disk errors. When the power is abruptly cut off, ZFS recovers as
though nothing went wrong, no long fsck process needed. ZFS snapshots
and zfs send | zfs recv make backups fast and efficient. OpenBSD's
lack of ZFS support became a dealbreaker for me when I had frequent
kernel panics due to a hardware failure in my server. OpenBSD would then
fsck my disks, which could take upwards of a half hour due to the
sheer volume of data that I have. Even after correcting errors, I still
had quite a scare with my database—it appeared corrupted and was
causing my Matrix homeserver to break. I was able to recover by
performing a VACUUM, which rebuilds some of the cluster files,
I suppose, but had the filesystem been consistently brought back
online after the power outage, that would not have happened. Or, even
if it did, ZFS would have allowed me to just roll back to the last
snapshot instead of having to connect an external drive with an
rsync-ed copy of the database on it.
So, knowing I needed to use ZFS, I chose to switch to FreeBSD. I looked at FreeBSD first because it has had ZFS support for ages. ZFS is a first-class citizen in FreeBSD, which means it can be used on the root filesystem, and be fully encrypted with FreeBSD's disk encryption framework. I didn't even consider Linux because I assumed that ZFS support, being a much more recent innovation, would not be as stable. I suppose that I was also afraid that Linux would be too unfamiliar, since it has been a long time since I've used it. I wanted something that closely resembled OpenBSD, so I assumed another BSD would be the way to go.
My journey on FreeBSD has been a rocky one so far, so much so that I don't wish to continue it. I more or less just expect my operating systems to work right out of the box. I generally don't modify systems heavily, so I expect that they will just run smoothly on their own. Now, this doesn't mean they need to be complete out of the box—I've long given up on trying to use a base install without external packages because it simply doesn't meet the demands of the modern internet—but it does mean that the hardware I'm running my operating systems on should just work, and the upgrade process and general system maintenance should be easy and work as expected. This was absolutely the case with OpenBSD, but I've found that neither of these are the case with FreeBSD. While it has ZFS, it has little else. I thought that ZFS would justify FreeBSD, but it simply has too many other pain points.
I installed FreeBSD 13.2-RELEASE a few weeks before 14.0-RELEASE was made available. As such, I quickly got to experience the FreeBSD upgrade process. Coming from the OpenBSD upgrade process, which is perhaps the smoothest upgrade process I have ever dealt with, the FreeBSD upgrade experience is disappointing, to say the least. The reason for this is simple: it took too long. I started the upgrade at 10am on a Friday, and it was not done until at least 16 hours later. I had to let it run overnight and then finish the remainder of the upgrade steps Saturday morning. This is simply unacceptable because it leaves the system in a very inconsistent state for a very long time, which I don't like. I posted to the forums and received no help. A few people said they had the same issue, and a few others said that something probably went wrong, although I have yet to find evidence of an error in the upgrade process. As far as I can tell, it worked fine, it just went really, really, really slow. And nobody can tell me why beyond some vague comments about "ZFS taking a long time with a lot of files". This is such a shame, because I want to like FreeBSD, but I just can't for a production server if it's going to take so long to upgrade. No other operating system I have ever used has taken so long to upgrade and provided so little output indicating what it was doing. To this day, I have no idea what could have possibly caused the upgrade to take so long. ZFS or no ZFS, upgrading from one OS release to the next should be no more than a 1 hour affair.
The other problem I have with FreeBSD is hardware support. While this isn't a huge deal for my server, which is quite old and thus well-supported all around, it is for my laptop. I want to run the same operating system on all of my hardware, server and laptop. This means that I need a general-purpose OS that is functional both as a client machine and a server. So not only does the operating system I use need to have good software support for the daemons I run on the server, it also needs to have good hardware support for my laptop. My laptop is a Framework Laptop (12th generation Intel), so the bar is not super high there, because it isn't like this is obscure hardware. In fact, the Framework is designed specifically for open-source operating systems. Still, FreeBSD disappoints here. I'm honestly quite surprised at this because OpenBSD is a far smaller and obscure project, yet it had absolutely perfect hardware support for my laptop. Everything simply worked, with the exception of the fingerprint scanner and Bluetooth, things that I never used anyway. I never had to mess with drivers either.
FreeBSD, on the other hand, does not support my laptop CPU's integrated graphics. I found this out the hard way after installing FreeBSD, installing the graphics driver, and then finding that the kernel panics when it loads that graphics driver. I verified on the forums that this is indeed the normal state of things for my CPU on FreeBSD. This means I can't use it with my docking station and I'd have to use the EFI framebuffer—which is not hardware-accelerated—if I wanted to use any graphical software. And of course that means no brightness control. That simply won't work for my laptop. If this were a stationary machine, that'd probably be fine, but I dock and undock my laptop frequently, using it sometimes as a standalone machine and sometimes connected to monitors, a keyboard, mouse, and other peripherals.
And so it appears, after a brief few-year hiatus to dive into the world of the BSDs, I am back on Linux. Linux has full hardware support for everything I can think to run it on. It is also the most well-supported in terms of software and online help. It has to my knowledge none of the problems that FreeBSD has, and it has support for ZFS, which is the only thing that OpenBSD lacks. In this way, it is the best of both worlds.
However many years ago it was when I switched to OpenBSD from Arch Linux, I did
so because I could not figure out how to get Linux to act as a router,
and I wanted an easy firewall. OpenBSD kept coming up in my searches
for a good server operating system that has good firewall and routing
support. Indeed it does; pf is a spectacular firewall, and all of the
network daemons required for a router are built-in. I think, at the time,
I was just scared of Linux's iptables. The syntax was confusing and
cryptic. I didn't understand the example configurations I read online. So
I wanted something simpler, and OpenBSD delivered in that area. I've
grown a lot in my understanding of networking and operating systems
since then, so I think I can tackle iptables again. It actually looks
like I can port my existing firewall and router configuration over in
fewer lines of text than I required for pf or FreeBSD's ipfw, which
is surprising to me. Having had to learn pf and ipfw, iptables
doesn't strike me as nearly as cryptic as it once did.
Ultimately, I want the simplicity of running the same operating system everywhere. My quest through the various UNIX-like operating systems out there has always been for that goal. I want the same OS on my server that's running on my laptop. This allows me to swap drives easily between the two if necessary, and allows me to gain a deeper understanding of my operating system because I can just focus on learning one system, instead of having to keep track of more than one. It makes upgrading simpler much more consistent, because I only have one OS to upgrade. It makes mirroring my software easier, because I only have to mirror the installation images and packages for one OS. Using the same operating system everywhere, even for different purposes, significantly decreases my mental load, which is always a good thing.
As much as I want to support the BSDs for a number of ideological reasons, the truth is that Linux is just far superior as a general-purpose operating system, in terms of hardware support and software functionality. It works equally well on a laptop and a server, and it supports all of the essential software I need, such as ZFS. No other open-source operating system can do everything that Linux does as well as Linux does it. They either have good software support, but bad hardware support (FreeBSD), or amazing hardware support but lacking support for modern software like ZFS (OpenBSD). Linux has both, which is why I have inevitably ended up back on it for my primary operating system.
For the same reasons I want to use the same operating system everywhere,
I also want to use the same distribution everywhere. Coming from OpenBSD
and FreeBSD, it should be no surprise that I chose Alpine Linux as the
distribution to run on my server and my laptop 1. Alpine is popular
for its use in virtual machines and Docker containers, but it also works
great on bare metal. I chose Alpine Linux because in many ways, it reminds
me of the BSDs. It is a small, simple operating system that doesn't use
systemd like most of the other Linux distributions, which means the way
services and networking are managed closely resembles other non-systemd
operating systems, such as the BSDs. In other words, Alpine Linux just
feels familiar as a BSD user, and only in part because it doesn't use
a GNU userspace.
I really appreciate Alpine's simplicity and minimalism. It allows me to build my own system with only the components I need. Most other Linux distributions are hefty, bundling in a lot of random packages that I just don't need or want. That's one of the appeals of the BSDs, is that they are comparatively simple systems compared to most Linux distributions. So having a lightweight Linux such as Alpine is comfortable for me.
When choosing a Linux distribution, I was most concerned about choosing one that would support an encrypted ZFS root filesystem. FreeBSD's own installer can set up such a system for you, and it is in fact the recommended configuration. ZFS on Linux, however, is a bit newer. As such, most installers don't seem to support it, however, it turns out that Alpine does support ZFS as the root filesystem with ZFS native encryption. FreeBSD configured encrypted block devices and then configured ZFS on top of them. The recommended configuration for Linux, however, seems to be to have ZFS on the disks themselves, and if you want encryption, using ZFS native encryption. I'm not quite sure how I feel about this yet—on the one hand, I appreciate that keeping ZFS on bare metal will allow me to connect my disks to any system that supports ZFS, Linux or not. On the other hand, this setup inherently leaks information about what sorts of files I have stored on my disks, even if they are encrypted. This is because ZFS encryption is at the dataset level, not at the disk level. So, if someone were to gain access to my drives, they would be able to at least list all of the datasets on the drive, even though they couldn't see the files inside.
I think that is a tradeoff I am willing to make. I'm very public about what kinds of services I run and what data I have, so as long as the contents of that data is safe, I don't think an attacker would get any more information about my system that I have not voluntarily shared. I trust the security of Linux. It has the most eyes on it because it is the most widely used. As such, bugs and vulnerabilities are regularly fixed. Additionally, the simplicity of Alpine works in my favor, because it requires me to install and configure everything I wish to run—there are few services running out of the box, so there are fewer attack vectors.
It also runs on my phone too, thanks to PostmarketOS! Finally, perfect OS consistency across all of my devices. I had that dream for OpenBSD, but it is actually a reality with Alpine Linux. ↩
© 2019-2024 Jordan Bancino.