My hobby? Speculative Engineering

I was thinking about my Famlab project this morning—the "homelab for busy families" idea where you reclaim data ownership (dare I mention #TechSovereignty?) without becoming a full-time sysadmin. Specifically, I was thinking about the maintenance of the base system and all services: the update path. It is an old idea that spawned from A/B updates in Android (where you have a second partition on which you install system updates, and then try to boot from afterwards; if it fails, you mark it as such and reboot into the other, older system partition) and the appeal from immutable operating systems—reproducible, easier deployments, safer.

How realistic is it to fully automate the testing of updates?

Imagine an automated testing pipeline that runs before every OS and service update. Say you're hosting Immich for your (family members') photos. An update comes out. The system spins up a temporary VM, loads it with representative test data (photos, albums, face detection samples), runs end-to-end tests to establish baseline metrics, proceeds through the upgrade, runs the suite again, compares results, and either proposes the upgrade or flags it for manual review.

We could even have a combination of pattern matching with some local LLM look at the changelog for "breaking changes" and other funny things like dependecy managements, and manual migrations.

The whole flow would be automatic, triggered as often as you want. The output would be a clear signal: update, or take a look. That would be grand, no effort spent in update management and a cool system to boot!

Taking a step back

Now, I don't actually have my Famlab running. I just have a collection of one-off services "for testing" that don't even have backups or reproducible setups. I am not even relying on Immich for my photos, today I upgraded it after not using it for six months. I have no idea if these updates break often. But I do know that Debian does not often break after an update.

What I have is an imagined future problem ("the update messed the whole thing up") and the (strong!) engineering impulse to solve it now, preemptively.

This is speculative engineering. And it looks like it's my hobby.

The pattern is really attractive. Start with a real goal (homelab for families), imagine future challenges (updates will break things), design sophisticated solutions (automated testing with VMs for both system and services), and then spend all your time building infrastructure for problems that don't exist yet. Meanwhile, the actual product never ships because you're too busy perfecting the deployment pipeline. I've done this before. You probably have too. (We use Continuous Discovery at work as a framework to avoid that, it is useful.)

What's actually the problem?

Spending zero brain celles on maintenance for Famlab means:

Updates do not break the system/service;
When updates do break something, they get rolled back automatically;
When updates degrade something, I have to step in and manually rollback / restore.

That's my actual constraint. This is not my job, and I don't do it for downstream instances—just for the nodes I decide to manage for the people I care about. It's not a critical service. So in fine I do not need a full testing pipeline across heterogenous services with deep metrics.

The driving hypothesis behind Famlab is that it can be good enough to compete with proprietary/commercial/privacy-invasive offerings. But I don't have the actual pain that points towards fully-tested updates. Instead, I need something to get started:

Basic health checks such as: did the system boot? did the service start? does it respond?
System snapshots of the service and its data before any change (courtesy of the hypervisor)
Cheap/easy staging environment based on system snapshots
Automatic rollback on failure after staging an update

In other words, the initial problem I have is: "Don't let updates break production in ways that cannot be undone trivially". Having the option to rescue the system easily is the peace of mind I need for production.

Man, it'd be cool to build

That over-engineered pipeline would be really fun to work on. It's full of concepts I have a passing idea about and a perfect excuse to dive in. The end-goal, the perfect solution, would of course be amazing. And given the number of moving parts and requirements, there would always be something to tackle and thing about—never quite acknowledging the project is too big, too ambitious.

I like to make these plans in my head. That's fine, it's problem solving, and it's rewarding. I decided to write it down this time because there's a pattern here worth naming explicitly.

The engineering mindset that makes us good at solving complex problems also makes us prone to solving problems that don't exist yet. We see the future system, extrapolate its challenges, and start building solutions before we've validated that the challenges are real. It's a feature and a bug: the same forward-thinking that helps us design extensible systems can also trap us in perpetual infrastructure work.

The trick is knowing when you're planning for inevitable scaling challenges vs. when you're just procrastinating by building cool infrastructure.

Next time we are designing something sophisticated for an anticipated problem, especially for side-projects, we ought to ask ourselves:

What's the simplest thing that solves the problem I have now? Remember that it takes time and effort to make things simple.
What would I learn by shipping that first? Build up data points and confidence that will help you steer.
What would it cost to solve the problem later? Some debt is worth taking.
Am I building this because it's needed or because it's interesting? Differentiate the underlying problem vs. your itch to act on cool and flashy.

We will still need, as always, to contextualise these questions and weigh the options. To me, that remains the core of the job, and why we end up calling ourselves seniors when we are able to do that.

Do not get infatuated with your one-off speculative engineering feats. Let them go. Small steps.