A private BSD laboratory

Yesterday we have completed the upgrade process from RELENG_7_0 to RELENG_7_1 on all our production servers. Many of these machine are “jail hosters”, acting as a sort of mini-mainframe, which are running several partitions of FreeBSD on the same piece of hardware. This has been the first time we upgraded FreeBSD in our production environment. We used the “compile-from-source” method, which needed the following steps:

1. sync sources from a local cvsup-mirror to the build host;
2. make binaries on the build host (once per architecture we are supporting);
3. mount /usr/src and /usr/obj via NFS on all systems to be upgraded;
4. run mergemaster in pre-buildworld mode on the upgrading host (once for the host and once for each jail with the -D flag);
5. install the new kernel on the upgrading host;
6. reboot the host with the new kernel in single user mode;
7. install the new userland for the host and for the basejail (we are using the ezjail framework);
8. run mergemaster on the host to align its configuration files to the new release;
9. boot into multi user mode;
10. run mergemaster with the -D flag to update each jail’s configuration files.

The real shutdown maintenance window that our users have actually felt  (steps 6 to 9) for the services running on a dual AMD Opteron 275 machine has been around 10 minutes. That machine was running 14 active jails. This means that the scheduled service disease for each server managed with the jail system was worth about 1:40 minutes!

Just one question: how much time do you think we would have to spend if we had chosen to manage our workloads with a cutting-edge virtualization infrastructure, such as Microsoft Hyper-V? 😉