A private BSD laboratory

I always had very little time to spend in scripting, but the need to have a “near-online” offsite backup for configuration files and data of several jails consolidated on a single FreeBSD host forced me to build this simple solution.

At first, I wrote a couple of rows to target a single specific need of mine but then, surprisingly, I was able to clean up the code so much to make it useful for a bit wider range of scenarios. Thus I think this little work might be useful for someone out there.

The idea of using rsync(1) to remotely backup data is not new… the only value added this solution brings is to make it as simple as possible. Anyone can use it to selectively backup configuration files and data on a per-jail basis.
Basically, now the “jailback” script leverage on the periodic(8) utility to build and maintain via SSH a remote copy of a set of selected folders for each jail (a jail administrator list them in the .jailback configuration file, in the root of the jail).

The advantages of using this script instead of a comprehensive data backup product (such as Amanda or Bacula):

• No need to setup, configure and maintain any backup agent on each jail.
• It has very little dependancies from code external to the base system (only rsync needs to be installed, and only on the host systems, not on every jail).
• This script is able to read the ezjail configuration file, thus barely integrating in your existing jail management framework.

Other important features, not exclusive of this solution:

• All data transfer between the source host and the backup server are encrypted.
• The amount of bandwidth required to keep synchronized the offsite copy is minimal, since rsync is able to transfer only the bits changed from the latest backup.

I creatively named this solution “jailback”. Here you can download the two files you need to give it a try:

230.jailback
You need to put this shell script  into /usr/local/etc/periodic/daily. When done, you’ll able to activate and customize the behavior of the script by defining the following variables in periodic.conf(5):

daily_jailback_enable (self-explanatory)

daily_jailback_dstdir
By default, all data is saved in a directory whose name is set to the hostname of the source machine, under /var/backups/ on the destination host. This variable enables you to specify a different folder name).

daily_jailback_srcdir
If ezjail(5) is installed on source host, by default the value of ${ezjail_jaildir} from the file ezjail.conf is used (then, it defaults to the folder /usr/jails). This variables allows you to specify the folder in which all your jails reside (symlinks are allowed).

daily_jailback_port
By default the script use the standard SSH port (TCP 22). If your backup server is only reachable on another port you can specify it here.

.jailback
You need to place this file in the root of each jail. In this way, if you delegate to other people the management of each single jail, they will be able to define what needs to be backed up without annoying you.

Notes

To have this method working, remember to test that you have the SSH daemon running on the backup host, to be able to reach it from the systems backed up and to have the key authentication method working correctly between the two.

It’s also desirable to protect data on the backup host with some sort of disk encryption method. Often this machine is located several kilometers away in a site without the same level of physical protection the main site has.

Updates

On march 28, 2011 I finally added a feature I needed since the first release of the script: it’s now possible to manage an exclusion list (for example, have all *.tmp files in a folder we want to backup excluded from the process). You can simply add a file named .jailback-exclude in the root of each jail where all excluded patterns are defined (see the options named “–exclude-from” in the rsync(1) man page).

The latest release of the script is always available here.

Today we completed the first major upgrade (from RELENG_7_2 to RELENG_8_0) in place on a production server in our corporate CED. This upgrade run quite smoothly, and has been useful to evaluate what are the critical aspects to care about during the upgrade process. Both the machines are “jailers”, running several instances of FreeBSD.

Actually, this has been the second upgrade: the first was done for one host in our disaster-proof external facility, a few days ago.
After this preparation, I’m feeling ready to upgrade the next “big” machine, which is running 21 jails in production on FreeBSD/amd64 7.2-p5.

UPDATE (Feb 27, 2010):

At the end of the day, the effort needed to upgrade these systems have been worth it: on our biggest “jailer” we spent only 6 minutes in upgrading each OS instance and, even counting how much time we need to reinstall all applications running in each jail, that’s far less time than migrating services from Windows server hosts to newer ones (the in-place upgrade would not ever been feasible on the Windows server platform!).

I’ve just read a very intriguing post by Ivan Voras titled “The night of 1000 jails” and I wish only report some reflections I’ve posted as comment to his article.

FreeBSD Jails are perhaps the best responses to today’s optimization needs in the whole IT world, and I’m really committed to make the most IT pros aware of this big opportunity to cut costs and increase efficiency, and reducing complexity at the same time.

I think that both network stack virtualization and ZFS support are essentials pieces which complete the technical picture, but we need a comprehensive management tool to convince IT staffs to adopt this great solution. I feel the best candidate to become such a definitive tool for managing jails in production business environments is the ezjail framework (http://erdgeist.org/arts/software/ezjail/), but it should leverage the latest features (FIBs, vimage, ZFS, etc.) as soon as they are available for each -RELEASE.

With such tools in the hands, I’ll definitely be able to make a bunch of sysadmins switch from the most famous hardware-virtualization tools to FreeBSD Jails, today!

My “Unix mainframe”, built on FreeBSD and the ezjail framework, has undergone its second minor upgrade, from the 7.1 to 7.2 release branch. I published this post some days later the upgrade because it did not come to my mind at that time… maybe I become considering this step an ordinary operation!? 🙂
In fact I’m not writing all what I’m experiencing around the BSD platform with the same frequency I did in the past, both because anything news I discover about this world is always better published on the web, and because I have not been experiencing any issue with this platform which is enough serious to make me write something about it.

Yesterday we have completed the upgrade process from RELENG_7_0 to RELENG_7_1 on all our production servers. Many of these machine are “jail hosters”, acting as a sort of mini-mainframe, which are running several partitions of FreeBSD on the same piece of hardware. This has been the first time we upgraded FreeBSD in our production environment. We used the “compile-from-source” method, which needed the following steps:

1. sync sources from a local cvsup-mirror to the build host;
2. make binaries on the build host (once per architecture we are supporting);
3. mount /usr/src and /usr/obj via NFS on all systems to be upgraded;
4. run mergemaster in pre-buildworld mode on the upgrading host (once for the host and once for each jail with the -D flag);
5. install the new kernel on the upgrading host;
6. reboot the host with the new kernel in single user mode;
7. install the new userland for the host and for the basejail (we are using the ezjail framework);
8. run mergemaster on the host to align its configuration files to the new release;
9. boot into multi user mode;
10. run mergemaster with the -D flag to update each jail’s configuration files.

The real shutdown maintenance window that our users have actually felt  (steps 6 to 9) for the services running on a dual AMD Opteron 275 machine has been around 10 minutes. That machine was running 14 active jails. This means that the scheduled service disease for each server managed with the jail system was worth about 1:40 minutes!

Just one question: how much time do you think we would have to spend if we had chosen to manage our workloads with a cutting-edge virtualization infrastructure, such as Microsoft Hyper-V? 😉

Maybe the last step of the migration process from Microsoft technologies to Unix BSD infrastructure was made last night. I say “mybe” because I’m sure a lot of work is left to be done (who can honestly say: “that’s the last step” until the facts show he was right?). By the way, today the last ISA Server web listener was tuned off and NGINX is proudly running and publishing all the web services behind our corporate redundant pfSense firewalls! 🙂

By now on, there is no public-accessible service provided by Microsoft’s technology in our business anymore. A couple of Domain Controllers and member servers are still serving some little LOB applications to our corporate users, but we are committed to migrate them as soon as possible to a new application framework (they are still running on Windows SharePoint Services).

The hard work that has been done during the last months seems to bring us some first thorough results, and I’m really pleased to view such a strong commitment has been successful! That’s a clear evidence the path we took is the right way to deal with Microsoft’s failure.

With today’s named and dhcpd activation on a new FreeBSD jail, all the most important IT services  in my corporate network (databases, web applications, network services, e-mail and collaboration) have been migrated onto the Unix BSD architecture! Surely a lot of work should still be done: inbound and outbound web proxing for example, is still made by ISA Server because the last Exchange virtual machine needs to be published by that, but NGINX and Squid have been successfully tested in another production environment during the last months.

I’m too tired to write a complete report showing all the advantages gained thanks to the new platform, but I can say that I’m particulary satisfied by performance, functionalities and manegeability of the integrated communications server (about whom I’ve just posted) and the DNS and DHCP services (which are able to manage the DDNS requirements of both Windows client and server machines).

Three days ago I finally put in production a new pfSense firewalling system. That solution take over the virtualized Microsoft ISA Server 2006 array, freeing up resources for moving all currently running VMs to the first Virtual Server node. That will let me to rebuild the second host machine with a new instance on FreeBSD 7 for amd64, which will become the “mainframe” container for most of the services delivered by my infrastructure (thanks to the Jail OS virtualization system).

Leveraging the well known OpenBSD PF technology, built on m0n0wall and FreeBSD platforms, pfSense seems to be a very good solution if you need a reliable and fault-tolerant firewalling system at affordable price, since it’s released under open source license terms. In this case I’ve only invested some hours for studing the best deployment method and some money to buy two Soekris net5501 boxes, which have been the natural choice since the moderate throughput managed by my IT systems. These appliances are getting me sleep good at night, thinking that for whatever reason a ISP link failure or an hardware/software problem will happen, there will always be the second node ready to seamlessy (statefully) make all IP traffic go on! 🙂

Obviuosly, pfSense alone won’t ever be able to replace all the L7 features ISA Server was giving me. The most notably absence is a good web reverse proxy system to make all HTTP/S requests made against different hosts (HTTP header names) flow to the corresponding internal web servers. Until I’ll have to publish RPC-over-HTTP Exchange services, I’ll maintain a one-leg proxy only ISA inside my net, but as soon as I’ll complete the migration to the CommuniGate Pro platform, I’ll replace it with an already proven and high performing NGINX proxy.

More than one year has past since I began exploring the opportunity of using Unix BSD in a business scenario and, against all my expectations to find some big weakness, I realized that there are actually many advantages in implementing this technology. To be exaustive, I must say that a lot of this advantage come from the weakness of the new Microsoft technologies, whose aspects can be summarized in the slogan “do less with more”, and which has made me looking for something to be evaluated as good replacement! 🙁

After a lot of implementations of BSD systems both in test and production environments, today I began the first “true pilot” process for migrating from Microsoft to Unix (urgh… I didn’t ever believe to write these words!). A near-production Windows server, Intel x86 dual processor with 1 Gb of RAM and a couple of volumes based on Ultra3 SCSI Adaptec RAID technology has been reinstalled using FreeBSD RELENG_7_0, on which I’m going to build a complete jailed ecosystem. It’s not the first time that I do this process of course, but that’s the first one I put in action in my network, using a “real server” hardware.

The perspective to run more that ten jails on a single host (whose hardware is actually eight years old!) makes me thinking that the goal seems really to be reacheable now: “do more with less”! 😉

A lot of time has past since my last post… in any blog of mine, I never had a lot of time to write, but during the last months I’ve been even more busy (if it was ever possible!).

These few lines only to mark that I’ve finally took the time to upgrade my BSDBOX03 server system to the new FreeBSD version (6.3-p2) for i386. I opted to the source upgrade method, by downloading the RELENG_6_3 source files via CVSup, compiling the userland and kernel binaries and installing them (first for the host, then for the jailed systems).

An interesting particular: the mergemaster(8) step, to which I did never give too much attention, which helped me to update the configuration tree (all files under the /etc) even on all the jailed machines, by using the “-D [path-to-jail-root]” switch.

Only one strange result: all the man pages report the “FreeBSD 6.2” version in the footer space… I’ve just reinstalled the man page ditribution and obviously rebuilt the whatis database, without resolving this little issue… anyone out there who has experienced the same behaviour?