A private BSD laboratory

It’s been some days since I seriously began to look at the ways to keep my FreeBSD machines up to date: the 6.3-RELEASE is coming and I don’t want to be caught unprepared! 🙂
At this time I’ve tried almost all the upgrade procedures from 6.2-RELEASE to 6.3-RC1, both by using binary and source upgrade methods.
I’ve been happy discovering that it’s possible to go through a simple in-place upgrade in a few minutes without having to maintain the whole CVS repository, by using sysinstall(8), and I found very interesting to track the errata branch with the freebsd-update(8) tool. I’ve also realized that upgrading from Source is the best choice to maintain up to date both host and guests in jail environments and to track the -STABLE branch on every machine I need to.

The best tool I’ve ever seen.

When, some days ago, I come back to the FreeBSD ports collection looking for any package related to the jail system, I took the time to try this little framework build by Dirk Engling to manage with the least administrative effort a group of jails. You might not to trust me, but it’s the first time in my work life (at least, as I can remember) that I found a solution which seems not to have any defect, a piece of software built to address very specific needs, a wonderful tool which does not add any complexity to the target system and at the same time accomplishes completely and with extraordinary performances the goals it was born to get.

Actually the ezjail tools are only few very simple shell scripts working together to make it easier, faster, dependable and safer to manage a collection of jailed systems. It is exactly what should be any IT solution: simple and effective, without the arrogance to do whatever it is not addressed to (it does its job by relaying on the rc jail system script).

It’s a rare case to make me spend such enthusiastic words about anything built by a human being… 🙂 Thanks a lot to Dirk, and to any other who contribute with their work and their expertise to develop a better IT!

Some days has passed since I first tried the Jail partitioning system, moving my first steps in the field of “FreeBSD virtualization”. As soon as I built my first jailed machine, I felt the importance to flash in the mind the path I walked to reach the goal, thus (after some tests and nightly hours spent on this) I’m able to write down my first “how-to” blueprint, containing all the best practices I’ve experienced so far about creating a jailed system.

Obviously it cannot be considered a good guide to manage a jailed environment, mainly because of both experience and knowledge that I’m still missing. Actually, there are a lot (and surely they are the most) of things I don’t know about BSD system administration, and you should consider this short guide as the first work of a novice that want to understand how the process of jailing a FreeBSD system can be accomplished nicely with the least administrative effort… of course, as it should be today and with my current knowledge!

I’ve sorted all the operations I made in five sections:

• host system setup;

• host configuration;

• guest system build and configuration;

• guest image creation and deployment;

• guest image maintenance.

I know it looks like a MOF checklist! Please be patient: give me the time I need to get in confidence with BSD… after all, I’m still a MCT! 😉

Host system setup

I’m not the right person to tell you what is the best method to install a BSD server… I can only remind not to install anything that is not strongly needed by your server’s mission: I found that installing the “minimal” distribution set can give you a good start point if you are a novice like me. At this step, you might find useful reading the article “Building a Unix Server” by Dru Lavigne.

Host configuration

After our physical machine is up and running, we must modify the file /etc/rc.conf by inserting all IP aliases we would be able to assign to guest machines. The syntax to be used is showed by the following example:

ifconfig_xl0_alias0=”inet 192.168.1.2 netmask 255.255.255.0″ ifconfig_xl0_alias1=”inet 192.168.1.3 netmask 255.255.255.0″

After this, we have to change the behavior of some system services on the host, mainly in order to avoid conflicts that some network services running in the guests would cause by trying to listen concurrently on the same TCP/UDP ports, first in the rc.conf:

syslogd_flags=”-b bsdhost“    # where “bsdhost” is the host machine’s name

then, in the /etc/ssh/sshd_config:

ListenAddress 192.168.1.1    # the IP address of the physical machine.

Guest system build and configuration

Now it’s the time to build the first jailed system by issuing the “make world” statement. I don’t have the arrogance to write down this procedure by my own, since it is excellently described in the paragraph 15.4 of the FreeBSD Handbook: “Creating and Controlling Jails“. You’ll also find there the changes which need to be applied to the rc.conf file on the host system.

Obviously, we need to configure the jailed system as we are used to do for a physical FreeBSD machine, so we should spend a few minutes by setting up the DNS resolver, the startup daemons, and any other thing we need to have before starting the jail. In addition to these configurations, there are a couple of things to remember when customizing a jailed machine. For a generic approach, you can read another good explanation of the process of creating and configuring a jail under FreeBSD 6 in this article. Personally, I’ve found useful to write this in my rc.conf:

defaultrouter=”x.x.x.x“    # the only network parameter needed by the jail sshd_enable=”YES”        # if you wish to control the jail from remote sendmail_enable=”NONE”    # add this if you do not want a MTA syslogd_flags=”-ss”    # disable network access for syslogd.

Obviously, this is only my personal preference, and you may feel better with other options.

Guest image creation and deployment

Once you have your first jail up and running, you realize that if you would even add another jailed system you’ll have to step into the same procedure you’ve just completed. Probably it isn’t what you want, especially if you began playing with jails to be able to deploy new systems with the least administrative effort. To achieve this, we can easily make a “snapshot” of the jailed directory tree by using the tar command, immediately after the build process:

cd /usr/jail            # the directory containing all jails
tar jcvf snapshot.tar.bz2 www    # where “www” is the source jail name.

By this way we’ve saved a compressed image of the directory tree of a clean FreeBSD jail, which will turn to be useful whenever we’ll want to deploy a new jailed system: we’ll only have to untar the archive:

cd /usr/jail            # the directory containing all jails tar xvf snapshot.tar.bz2    # be careful not to overwrite the source jail.

Remember to set timezone and root password before start the jail with the /etc/rc.d/jail script, by issuing the following commands in the host’s shell:

jail [path] [hostname] [ipaddress] /bin/sh
tzsetup
passwd
exit

As we wanted, the process of deploying new systems has become a matter of few minutes (the most time is needed by the extraction process). Obviously there are some considerations we must take care of before put it in a production environment; we’ll see some of these in the next section.

Guest image maintenance

Not all systems are equal, and probably we’ll have to deploy a lot of jails to accomplish different server tasks, each needing different system configurations and packages. The best way I’ve found since now is to decompress the snapshot to be modified, apply all changes in the jail’s directory and then create a new compressed image by using the method I wrote in the previous paragraph. Be careful not to include any machine-specific setting (such as hostname, SSH key pair, etc.) before creating the image, to avoid potential problems when we’ll start a jail using the new snapshot.

This little guide come from my personal experience in working with the FreeBSD jail system and it cannot be referenced as a technical guide to build and maintain jails in a production environment. I’ll be glad if anyone more experienced and skilled than me on this theme will suggest how to make it more technically valuable and useful to FreeBSD systems administrators.

Yesterday I got the time to build a partitioned FreeBSD installation into a sandbox that we call “Jail”. This logical partition in which a BSD system can run quite independently from other instances on the same physical host seems what Sun Microsystems calls “Solaris Containers”: all processes running in the jail are not allowed to interact with the programs of the outside world, neither access the file system of the host or the others jailed machines. With this type of operating system-level virtualization all guests rely on the kernel of the host OS to run at the physical machine speed. Obviously we cannot gain the same isolation that a fully virtualized guest can achieve with Virtual Server, Xen or VMWare, but the performance of the guest are amazing and if the jail’s level of isolation is enough for you, this solution is one of the most exciting in the world of BSD operating systems: once I’ve “build the world” for the first jailed system and I’ve made a compressed tarball of its file system (about 52 Mb), I became able to set up any number of new BSD servers in about 3 minutes for each!!

A good start point to learn about the Jail system is obviously in the Handbook, but you can find other useful information all around the Net (I’ll post new links as soon as I’ll find some interesting ones):

• Architecture Handbook – The Jail Subsystem
• Sys Admin – Securing FreeBSD Using Jail
• The FreeBSD Diary – Jails under FreeBSD 6
• ONLamp.com – Virtualization with FreeBSD Jails

I’ve found some interesting articles by searching on the Windows Live Search engine.
You can also find some useful tools to manage jails in the sysutils section of the Ports Collection.

I’ve learned one more thing about Unix shell administration. In the past days I had to launch a few ‘make’ processes on a remote host via a SSH terminal, and each time I had to wait (keeping my Windows XP client running) until the process completed because I didn’t know any way to make the process running after I leave the terminal. It was becoming a huge handicap for me: I’m used to administer a lot of Microsoft Windows server machines all around the globe simply by creating, disconnecting and reconnecting a lot of RDP sessions where all processes I launch live until my logout… so I decided to spent some time searching a solution like that in the Unix field.

I only found the possibility to suspend and run in the background multiple simultaneous processes into the same terminal session, but soon I realized that it does not solved my issue, since all the processes were always killed as soon as I get disconnected from the SSH virtual console. Thus I posted my issue in the comp.unix.shell Usenet group, and after some days I get the definitive solution:

the SCREEN utility!

You can get it from the port collection under sysutils/screen, and is a very nice piece of software to free your commands from the risk of a network connection break when you are working on a virtual terminal to administer your Unix host.

Screen capabilities go far beyond the attach/detach process. You can read a quite complete guide to use and configure screen here.

Sometimes we learn a bit only after a lot of studying… but after such a commitment, that little thing can make us happy as if we discovered a new world!
Yesterday I finally understood the way to have the pkg_add utility grabbing the binary packages from a source other than the default for the system’s version release. It resulted to be a great saving of time in the process of installing a lot of useful piece of software from the FreeBSD ports collection, thereby preventing to waste your patience by searching, downloading and installing each latest revision of dependent/depending packages you need.

It was only necessary to set the PACKAGESITE environment variable to drive the behavior of this utility, by specifying in the value the full path to the version branch of the ports collection maintained by the FreeBSD technical staff. So, if you wish to install the latest revision of all software packages built for the 6.2-RELEASE, even if you are using the 6.1-RELEASE, you only need to issue this command in you shell: “setenv PACKAGESITE ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-6.2-release/Latest/“.

Did you read my previous posts about RcptFilter? After having put in production the third BSD server running BIND and Sendmail with ClamAV, I decided to extend a bit the central management configuration features of that tiny scripting solution. At scheduled times, few Perl and MSFT shell scripts running on a Windows Server 2003 machine produce and publish on a public web location a compressed archive containing all configuration files we need to distribute on all our BSD managed servers.

Some troubles occurred in generating the configuration files, because the Microsoft text file format is a bit different (each line ends with CR and LF characters instead of using LF only, as all Unix-like machines do), but with the utility dos2unix all came back compliant.

Considering the settings we need to centrally manage for our few servers, this solution fits well until now, but I wish extend it in order to distribute even some machine-specific settings, maybe using authenticated and protected channel to the hosts and a central repository of policies. Looking also forward to find a good solution which, I am sure, it has already been developed by good Unix administrators out there.

The first major release of the multi-user edition has been released several weeks ago. Only for testing purposes, the Phoibos project team members have begun the testing phase on the same BSD system used by the production Phoibos blog hosting service. The new collection of blogs is available at http://wp.valsania.it. Feel free to register and contribute to all other beta testers, here!

On the last week-end I’ve decided to migrate both my SMTP front-end and public DNS servers to the new BSD platform. This implementation lets me able to run the latest version of BIND and Sendmail, thus enabling an higher level of security. In addition, the latest version of ClamAV-milter is now checking all incoming messages at the SMTP connection layer, this is important for me to get rid of a large amount of malware which has been filtered out at the next levels (usually by the heuristic spam filter), until now.

I found that ClamAV can be a good solution to fit the needs for securing the SMTP traffic and for reducing the administrative overhead brought by the daily checking of the UCE folders on my back-end Exchange systems: the mind-free quarantine management (if the scanner finds any malware, it can immediately break down the current SMTP session, even without taking care of NDR generation) and the auto update process make it a plug-and-forget solution… at least, until there will be any trouble to resolve! 😉

Yesterday night I spent more than five hours on a FreeBSD 6.1 system in testing the clamav-milter service. Do you remember my last post, in which I wrote about the need to migrate all my SMTP front-end systems to the FreeBSD platform in order to achieve a better security check on incoming e-mail traffic? Now I’m able to run a Sendmail daemon tailored to my corporate needs which works together with a good (and free) virus scanning engine, dropping infected e-mails at the SMTP connection stage without generating any NDR (Non-Delivery Report) to the sender address. Also the reporting feature (an e-mail to the postmaster mailbox) in case of virus detection fits well in my environment, where there is no scheduled notification sent to the postmaster address.

I had to work a lot to find the best installation method for my production environment: after I built all the packages needed by ClamAV (by inserting the right flags during the make process) I was able to setup the latest version of the product on the production systems without installing the ports collection. Now I want to observe it running for some days before migrating the primary SMTP front-end system too.