About ten days ago, I received a scary e-mail from the NAS (it has a name by the way, it's called cactus): my boot pool was degraded but still operational. The SSD I added not so long ago to mirror the USB drive actually started to fail. TrueNAS detected slow read performance, and eventually detached the drive from the pool of its own accord. Smarty pants.
I was certainly not expecting the SSD to fail before the USB drive (they were roughly as old), especially given that the SSD did not see much wear and that it was stored in an ESD-safe bag together with a desiccant bag.
I ordered two more SSD drives to replace the current USB drive and the failed SSD, with the intent of switching to a full SSD mirrored boot pool. I also got a USB-to-SATA converter to connect one of the drives to the internal USB port currently used by the USB drive since I only have access to one internal SATA port (remember, the one for the ODD).
The plan was simple: connect one SSD first to the SATA port, attach it to the boot pool, let the system resilver the pool, detach the USB stick, power-off, swap the USB stick with the second SSD thanks to the USB-SATA adapter, boot on the first SSD, and do the attach-and-resilver dance once more with the second SSD. It should have been a fairly easy and painless process according to the documentation and online discussions. Except that it was not.
The first problem I encountered was of a surprising nature. Trying to attach the SSD to the boot pool gave the unhelpful error of "can only attach to mirrors and top-level disks". This is a known UX problem where a few type of errors are aggregated into this high-level exception thrown by TrueNAS' middleware. It is apparently quite common to see this when the block size of the pool and additional disk cannot match. I suspect the initial boot pool was created with a too-small block size (think 512 bytes) while now the recommendation is to go with 4 kilobytes by default. And since the new SSDs have indeed 4k block size, I cannot mix and match them in the existing pool. I did try to make it work for a couple of hours but eventually decided that my mastery level was too low to risk the NAS setup and the time investment needed not worth it. (Not that I would have had the time anyway.)
After reading extensively on how to migrate the TrueNAS boot drive to another medium, I jumped on the fresh install bandwagon. I installed the version I was currently running on the USB stick on the SSD, booted, imported the configuration, and done. It was really simple and it definitely took me longer to read about the procedure (a case of cold feet) than to execute it.
Now, the idea was to keep a mirror of the boot pool. I wanted to part ways with the USB stick for good and simply use two SSDs. Strangely enough, the second SSD via USB stopped working the moment I installed it inside the NAS. (I did run a couple of tests outside the NAS first, using the external USB ports.) The kernel reported I/O errors and simply backed away from the construct. But only on FreeBSD, I could use the SSD on a GNU/Linux machine without issues.
This time around, I did not spend too long thinking about the solution: the adapter goes back as DOA and I simply don't mirror the boot pool. Replacing the boot drive is so easy that I am not afraid of the next failure. I have only so much time to debug weird hardware problems. So for now the USB stick remains plugged in, but as a backup boot disk (it is not part of the running boot pool) — that way there is no write on it until the day I need to rescue the system and use it as the boot drive.