A private BSD laboratory

It’s been some days since I seriously began to look at the ways to keep my FreeBSD machines up to date: the 6.3-RELEASE is coming and I don’t want to be caught unprepared! 🙂
At this time I’ve tried almost all the upgrade procedures from 6.2-RELEASE to 6.3-RC1, both by using binary and source upgrade methods.
I’ve been happy discovering that it’s possible to go through a simple in-place upgrade in a few minutes without having to maintain the whole CVS repository, by using sysinstall(8), and I found very interesting to track the errata branch with the freebsd-update(8) tool. I’ve also realized that upgrading from Source is the best choice to maintain up to date both host and guests in jail environments and to track the -STABLE branch on every machine I need to.

The best tool I’ve ever seen.

When, some days ago, I come back to the FreeBSD ports collection looking for any package related to the jail system, I took the time to try this little framework build by Dirk Engling to manage with the least administrative effort a group of jails. You might not to trust me, but it’s the first time in my work life (at least, as I can remember) that I found a solution which seems not to have any defect, a piece of software built to address very specific needs, a wonderful tool which does not add any complexity to the target system and at the same time accomplishes completely and with extraordinary performances the goals it was born to get.

Actually the ezjail tools are only few very simple shell scripts working together to make it easier, faster, dependable and safer to manage a collection of jailed systems. It is exactly what should be any IT solution: simple and effective, without the arrogance to do whatever it is not addressed to (it does its job by relaying on the rc jail system script).

It’s a rare case to make me spend such enthusiastic words about anything built by a human being… 🙂 Thanks a lot to Dirk, and to any other who contribute with their work and their expertise to develop a better IT!

Some days has passed since I first tried the Jail partitioning system, moving my first steps in the field of “FreeBSD virtualization”. As soon as I built my first jailed machine, I felt the importance to flash in the mind the path I walked to reach the goal, thus (after some tests and nightly hours spent on this) I’m able to write down my first “how-to” blueprint, containing all the best practices I’ve experienced so far about creating a jailed system.

Obviously it cannot be considered a good guide to manage a jailed environment, mainly because of both experience and knowledge that I’m still missing. Actually, there are a lot (and surely they are the most) of things I don’t know about BSD system administration, and you should consider this short guide as the first work of a novice that want to understand how the process of jailing a FreeBSD system can be accomplished nicely with the least administrative effort… of course, as it should be today and with my current knowledge!

I’ve sorted all the operations I made in five sections:

• host system setup;

• host configuration;

• guest system build and configuration;

• guest image creation and deployment;

• guest image maintenance.

I know it looks like a MOF checklist! Please be patient: give me the time I need to get in confidence with BSD… after all, I’m still a MCT! 😉

Host system setup

I’m not the right person to tell you what is the best method to install a BSD server… I can only remind not to install anything that is not strongly needed by your server’s mission: I found that installing the “minimal” distribution set can give you a good start point if you are a novice like me. At this step, you might find useful reading the article “Building a Unix Server” by Dru Lavigne.

Host configuration

After our physical machine is up and running, we must modify the file /etc/rc.conf by inserting all IP aliases we would be able to assign to guest machines. The syntax to be used is showed by the following example:

ifconfig_xl0_alias0=”inet 192.168.1.2 netmask 255.255.255.0″ ifconfig_xl0_alias1=”inet 192.168.1.3 netmask 255.255.255.0″

After this, we have to change the behavior of some system services on the host, mainly in order to avoid conflicts that some network services running in the guests would cause by trying to listen concurrently on the same TCP/UDP ports, first in the rc.conf:

syslogd_flags=”-b bsdhost“    # where “bsdhost” is the host machine’s name

then, in the /etc/ssh/sshd_config:

ListenAddress 192.168.1.1    # the IP address of the physical machine.

Guest system build and configuration

Now it’s the time to build the first jailed system by issuing the “make world” statement. I don’t have the arrogance to write down this procedure by my own, since it is excellently described in the paragraph 15.4 of the FreeBSD Handbook: “Creating and Controlling Jails“. You’ll also find there the changes which need to be applied to the rc.conf file on the host system.

Obviously, we need to configure the jailed system as we are used to do for a physical FreeBSD machine, so we should spend a few minutes by setting up the DNS resolver, the startup daemons, and any other thing we need to have before starting the jail. In addition to these configurations, there are a couple of things to remember when customizing a jailed machine. For a generic approach, you can read another good explanation of the process of creating and configuring a jail under FreeBSD 6 in this article. Personally, I’ve found useful to write this in my rc.conf:

defaultrouter=”x.x.x.x“    # the only network parameter needed by the jail sshd_enable=”YES”        # if you wish to control the jail from remote sendmail_enable=”NONE”    # add this if you do not want a MTA syslogd_flags=”-ss”    # disable network access for syslogd.

Obviously, this is only my personal preference, and you may feel better with other options.

Guest image creation and deployment

Once you have your first jail up and running, you realize that if you would even add another jailed system you’ll have to step into the same procedure you’ve just completed. Probably it isn’t what you want, especially if you began playing with jails to be able to deploy new systems with the least administrative effort. To achieve this, we can easily make a “snapshot” of the jailed directory tree by using the tar command, immediately after the build process:

cd /usr/jail            # the directory containing all jails
tar jcvf snapshot.tar.bz2 www    # where “www” is the source jail name.

By this way we’ve saved a compressed image of the directory tree of a clean FreeBSD jail, which will turn to be useful whenever we’ll want to deploy a new jailed system: we’ll only have to untar the archive:

cd /usr/jail            # the directory containing all jails tar xvf snapshot.tar.bz2    # be careful not to overwrite the source jail.

Remember to set timezone and root password before start the jail with the /etc/rc.d/jail script, by issuing the following commands in the host’s shell:

jail [path] [hostname] [ipaddress] /bin/sh
tzsetup
passwd
exit

As we wanted, the process of deploying new systems has become a matter of few minutes (the most time is needed by the extraction process). Obviously there are some considerations we must take care of before put it in a production environment; we’ll see some of these in the next section.

Guest image maintenance

Not all systems are equal, and probably we’ll have to deploy a lot of jails to accomplish different server tasks, each needing different system configurations and packages. The best way I’ve found since now is to decompress the snapshot to be modified, apply all changes in the jail’s directory and then create a new compressed image by using the method I wrote in the previous paragraph. Be careful not to include any machine-specific setting (such as hostname, SSH key pair, etc.) before creating the image, to avoid potential problems when we’ll start a jail using the new snapshot.

This little guide come from my personal experience in working with the FreeBSD jail system and it cannot be referenced as a technical guide to build and maintain jails in a production environment. I’ll be glad if anyone more experienced and skilled than me on this theme will suggest how to make it more technically valuable and useful to FreeBSD systems administrators.

Yesterday I got the time to build a partitioned FreeBSD installation into a sandbox that we call “Jail”. This logical partition in which a BSD system can run quite independently from other instances on the same physical host seems what Sun Microsystems calls “Solaris Containers”: all processes running in the jail are not allowed to interact with the programs of the outside world, neither access the file system of the host or the others jailed machines. With this type of operating system-level virtualization all guests rely on the kernel of the host OS to run at the physical machine speed. Obviously we cannot gain the same isolation that a fully virtualized guest can achieve with Virtual Server, Xen or VMWare, but the performance of the guest are amazing and if the jail’s level of isolation is enough for you, this solution is one of the most exciting in the world of BSD operating systems: once I’ve “build the world” for the first jailed system and I’ve made a compressed tarball of its file system (about 52 Mb), I became able to set up any number of new BSD servers in about 3 minutes for each!!

A good start point to learn about the Jail system is obviously in the Handbook, but you can find other useful information all around the Net (I’ll post new links as soon as I’ll find some interesting ones):

• Architecture Handbook – The Jail Subsystem
• Sys Admin – Securing FreeBSD Using Jail
• The FreeBSD Diary – Jails under FreeBSD 6
• ONLamp.com – Virtualization with FreeBSD Jails

I’ve found some interesting articles by searching on the Windows Live Search engine.
You can also find some useful tools to manage jails in the sysutils section of the Ports Collection.

I’ve learned one more thing about Unix shell administration. In the past days I had to launch a few ‘make’ processes on a remote host via a SSH terminal, and each time I had to wait (keeping my Windows XP client running) until the process completed because I didn’t know any way to make the process running after I leave the terminal. It was becoming a huge handicap for me: I’m used to administer a lot of Microsoft Windows server machines all around the globe simply by creating, disconnecting and reconnecting a lot of RDP sessions where all processes I launch live until my logout… so I decided to spent some time searching a solution like that in the Unix field.

I only found the possibility to suspend and run in the background multiple simultaneous processes into the same terminal session, but soon I realized that it does not solved my issue, since all the processes were always killed as soon as I get disconnected from the SSH virtual console. Thus I posted my issue in the comp.unix.shell Usenet group, and after some days I get the definitive solution:

the SCREEN utility!

You can get it from the port collection under sysutils/screen, and is a very nice piece of software to free your commands from the risk of a network connection break when you are working on a virtual terminal to administer your Unix host.

Screen capabilities go far beyond the attach/detach process. You can read a quite complete guide to use and configure screen here.

Sometimes we learn a bit only after a lot of studying… but after such a commitment, that little thing can make us happy as if we discovered a new world!
Yesterday I finally understood the way to have the pkg_add utility grabbing the binary packages from a source other than the default for the system’s version release. It resulted to be a great saving of time in the process of installing a lot of useful piece of software from the FreeBSD ports collection, thereby preventing to waste your patience by searching, downloading and installing each latest revision of dependent/depending packages you need.

It was only necessary to set the PACKAGESITE environment variable to drive the behavior of this utility, by specifying in the value the full path to the version branch of the ports collection maintained by the FreeBSD technical staff. So, if you wish to install the latest revision of all software packages built for the 6.2-RELEASE, even if you are using the 6.1-RELEASE, you only need to issue this command in you shell: “setenv PACKAGESITE ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-6.2-release/Latest/“.

Did you read my previous posts about RcptFilter? After having put in production the third BSD server running BIND and Sendmail with ClamAV, I decided to extend a bit the central management configuration features of that tiny scripting solution. At scheduled times, few Perl and MSFT shell scripts running on a Windows Server 2003 machine produce and publish on a public web location a compressed archive containing all configuration files we need to distribute on all our BSD managed servers.

Some troubles occurred in generating the configuration files, because the Microsoft text file format is a bit different (each line ends with CR and LF characters instead of using LF only, as all Unix-like machines do), but with the utility dos2unix all came back compliant.

Considering the settings we need to centrally manage for our few servers, this solution fits well until now, but I wish extend it in order to distribute even some machine-specific settings, maybe using authenticated and protected channel to the hosts and a central repository of policies. Looking also forward to find a good solution which, I am sure, it has already been developed by good Unix administrators out there.

Some days ago I had to setup a public DNS server (on FreeBSD) to act as primary name server for a couple of domains and (since I was there) to maintain slave zones for all my public domains.

I’m not exactly what you’d call a “Unix system administrator”, but I have some experience about configuring BIND since the 8.x family, so I was a bit disappointed when, after spending some time to accurately write down the named.conf, I read a “connection reset” error in the log file.

At first, the matter appeared to be related to somewhat resolvable by inserting the right statements in the config, because I noticed the following alert message from the Cisco IOS firewall IDS module at the location of the primary name server:
“%IDS-4-DNS_ZONE_XFER_REQUEST_HIGH: Sig:6052:DNS High Zone Transfer Request”.

Thus I combined all these statements:
[ query-source [ address ( ip_addr | * ) ] [ port ( ip_port | * ) ]; ]
[ transfer-source (ip4_addr | *) [port ip_port] ; ]
[ alt-transfer-source (ip4_addr | *) [port ip_port] ; ]

Nothing changed. So I tried to transfer the same zone using a different name server in the same network of my FreeBSD machine: it was a Windows Server 2003 host. By using the Microsoft DNS service (without changing anything at the network layer) all worked fine at the fist shoot!

By looking for someone who’d have ever had such a problem, I found a post in a bug tracker list. It was explained that there is no way (neither there will ever be) to have BIND sending zone transfer request to other name server from its port 53 TCP. All the statements I had tried apply only to UDP traffic, since all TCP request are always sent from a random high port.

Thus, the only applicable solution has revealed to be changing the Cisco IOS firewall behavior in such a way to not react if a request originating from known external DNS servers matches any ip audit signature… 100 point to subtract from my personal BIND scorecard! 🙁

I had setup a front-end SMTP relay system one more time, running on the latest release currently in the stable branch of the FreeBSD Project development (6.2).
I’m a bit experienced in running Sendmail on BSD systems, so I spent only few minutes to configure and test e-mail routing, and I took some time for having a look at a strange behavior that I’m not be able to understand yet.

Think as if the host name was mx2.mydomain.com: since I had to build it as a secondary mail exchanger for several email domains, after the insertion of all the domains in the relay-domains Sendmail configuration file, I wrote in the mailertable to have all messages sent to these domains relayed to the primary SMTP inbound server (maybe located in a different location), by inserting this lines:

With this config, delivery problems came for the messages sent to the first domain, whereas all mail sent to others domains were successfully relayed. The error messages that appeared in the log files were suggesting a clear matter: “stat=Deferred: Name server: mx1.mydomain.com : host name lookup failure”. This should not leave you any doubt, but after I checked there was not any name resolution problem, I found that a bit strange: in fact, why only delivering to the first domain was faulting?
I decided to change the mx1.mydomain.com name with its corresponding IP address, and all worked fine, for all domains!

After a lot of testing, I found that Sendmail seems not to be able to relay any message to the primary MX if its FQDN lies in the same DNS domain of the Sendmail host!! I posted this unbelievable matter in the comp.mail.sendmail newsgroup, but I haven’t got any reply yet… please someone show me that the problem is due only to an error in my configuration: I cannot believe there is such a serious bug in Sendmail without anyone has ever noticed it! :S

The first major release of the multi-user edition has been released several weeks ago. Only for testing purposes, the Phoibos project team members have begun the testing phase on the same BSD system used by the production Phoibos blog hosting service. The new collection of blogs is available at http://wp.valsania.it. Feel free to register and contribute to all other beta testers, here!