That said, we must not forget Ubuntu’s focus, which I think (and I will expand on it later) is still valid: desktop and laptop computers.

Ubuntu aims to bring the latest drivers and technologies to desktops (I will use desktop as a term for desktop computers, laptops, and netbooks here). It has to, because otherwise it won’t be able to compete with proprietary operating systems (read: Windows and Mac OSX). To achieve that, it has to put the GPL/non-GPL debate (which is a big issue for Debian et al) aside. There’s a bunch of repositories of not exactly free (or not even open source) software, which is essential to get certain hardware (e.g. graphics cards) and software (e.g. media codecs) working: Restricted, Multiverse, Universe, Medibuntu, etc. Although they are not officially supported, all of them except Medibuntu are included in /etc/apt/sources.list and active, plus they reside on *.ubuntu.com servers. So it’s a bit difficult to not consider them part of Ubuntu, or at least part of the Ubuntu-Conquers-The-Desktop success, which makes the discussion of “who’s responsible for what?” a bit more difficult to answer. But it’s a crucial question in an enterprise setup. This is just one example why I think that Ubuntu is not targeting enterprise server environments, and you can’t be the best choice for something, which you are not focused at. More further down…

I’ve just installed Ubuntu Server 10.4 LTS in a virtual machine here to verify whether my past experience still holds true. I went for the Install Ubuntu Server option, and only used defaults (except that I added OpenSSH). So except stated otherwise, I will refer to this version, which is the latest release for servers and allegedly targets enterprises.

The intention of this article is not to compare Linux distributions with each other or give any recommendations as to which Linux distribution is the best one to go for in an enterprise environment. It’s not my intention to badmouth Ubuntu or say that it’s not suitable for servers at all, either. I’m merely explaining why I’m not a big fan of Ubuntu, as I’ve been asked that question a couple of times recently. Okay, maybe I’ve provoked that question a little.  It’s no secret though that my favourite Linux distribution for servers is CentOS, if it has to be Linux, or FreeBSD, if the scenario permits and the operating system decision is a matter of what we want to achieve rather than what we want to use. But again, that’s a separate discussion and beyond the scope of this article. Also, there’s no “one size fits it all”. I have noticed that many people stick to the things they know or like best in many situations, where another operating system or Linux distribution might have been more suitable for a certain job. Although I can’t scientifically prove it, this seemingly applies to many people who use or did use Ubuntu on desktops. Maybe we should use the “fanboi” term not only for Apple’s repeat customers, but also for Ubuntu users

Okay, back to the original question, why I don’t like Ubuntu on servers…

Let me first define what my expectations are:

1. There is no such thing as one single server. Servers come in pairs at the very least. I build environments which are as fail-safe as possible (and affordable), load-balanced, robust.
2. Implementing the very latest developments and technologies usually does more harm than good, because they can’t have been tested by as many people as older features. I prefer well-tested, solid operating systems. If I really need a more up-to-date version of, say, PHP, then I build a package for that. I don’t need the entire distribution to include the latest features just because I need only one package to be a bit more up-to-date! (NB: I am talking about feature updates here, not security patches!)
3. I expect the operating system to provide reasonable security standards and default settings and leave the rest to me.
4. I prefer using established standard tools and best practices over “Mate, we’ve quickly put together a new tool for you”
5. I decide what is installed and what isn’t. I don’t need the OS to tell me what it thinks is good for me.
6. Most of all I expect a proper release cycle and thorough testing before labelling something as a final release. (Oh, I did mention that before, didn’t I? )
7. I don’t like operating systems or derivates, which are entirely built on top of an existing one. Additional layers cause additional dependencies, often inherit errors, and make it more difficult to track down where an error comes from, and who has introduced it.

Let me start with 7., because I hear you saying “But…”.  No, CentOS is not built on top of RedHat! It’s a 100% clone minus proprietary stuff, logos, and license/support costs. Ubuntu however is derived from Debian and has added loads of stuff, which includes many things that Debian refuses to include (e.g. proprietary drivers and non-GPL code in general), while incorporating lots of Debian packages. Remember this severe OpenSSL bug exactly two years ago? What happened was that Debian broke the random number generator (making keys predictable) in their OpenSSL package.  The only distributions affected were Debian and all derivates including Ubuntu, but not RedHat or clones/derivates thereof. I don’t blame Ubuntu for inheriting broken code, because nobody can possibly read and understand the source code of everything. However, that was when I lost trust in Debian (the code change was an utterly stupid attempt to get rid off compiler warnings without understanding what the code does), and as it is the foundation of Ubuntu, I can’t trust it either. You may call it nitpicking, but making changes (and introducing bugs) to crucial security related features, which would definitely not have gotten the upstream’s approval, if they had pushed it upstream, is pretty bad stuff. All SSH keys had to be re-generated and SSL certificates replaced. Not  a big deal for only a bunch of servers, but a massive amount of work for an enterprise.

Let me continue traversing the list above. Number 6: Releases. First of all, before installing a new update, I would like to be able to assess what changes will occur to my systems. That’s what release notes are for. However, if you are on the Ubuntu Server home page and click on Resources and then a bit further down on Release Notes, you in fact end up only with known issues for both Ubuntu Desktop and Server. It takes quite a while to find the actual key specs at least, hidden somewhere in the wiki. But I wanted to elaborate on release cycles…

From a server Linux distribution I would expect that it has been presented to a huge group of users prior to its final release. Ideally it goes through various beta or pre-release cycles, giving the users time to test (some things need time to test them properly) and developers time to fix issues. Ubuntu however sets deadlines: every April and October of each year, there’s has to be a major release. In other words: In a half-year cycle new features have to be selected, introduced, and tested. It doesn’t seem to be top priority to have rock-solid releases. Let me quote an Ubuntu developer:  ”And remember that, since this is a long-term supported (LTS) release, there are ample opportunities for further bugfixes after the final release by way of the SRU process[2]. Point releases for Ubuntu and Kubuntu LTS will be made at roughly six-month intervals, with the first expected in July 2010 to address any critical issues not identified or fixed in time for the 10.04 LTS release.”

I’m sure he didn’t mean it, but it sounds like: “Hurry up. Doesn’t matter if we can’t fix things on time, as we’ll come up with a bugfix release in July anyway.”  Beta 2, release candidate, and final release were published within only three weeks, by the way. Ubuntu, Ubuntu Server, and Kubuntu at the same time. It does raise questions, doesn’t it?

If you look at FreeBSD, just to compare two entirely different release policies, you’ll find that they first work out what issues need to be addressed and which features may be introduced. Then they come up with a very rough schedule. And then, after they have frozen the code, they go through many stages for major releases: BETA 1-4, Documentation updates, Release Candidate 1-3, Release. From the code freeze (except for bug fixes) to the actual release of 8.0 in November 2009, it took them 4.5 months. And, as usual, the result is a rock-solid operating system. The minor release 8.1 is planned for July this year (but not yet announced for a good reason). I expect it to be available in September or so. In my opinion, it’s much more important to get the issues solved rather than sticking to a fixed deadline.

Number 5: I’m the boss! As I said earlier, I just installed 10.04 LTS Server here. Although I did not select any packages except OpenSSH, I ended up with an installation eating 818 MB on my disk. Hello? It turned out that a whole pile of useless stuff is installed by default: Wireless support, PPP (yeah, good old dial-up!) support, file system support for NTFS and FAT32, tools to compile C/C++ etc. Seriously, that’s not funny. So the first thing I will have to do is remove all the litter (or scroll through lists of useless crap at install time and deselect there).

Also, I can’t remember that I have been asked whether or not I wanted AppArmor installed. I don’t! SELinux has been in the mainline Linux kernel since 2003. I don’t want that to be removed and replaced with another solution. At least I would like to have a choice. (However, I do embrace that Ubuntu comes with AppArmor now, which is still better than Debian’s and Ubuntu’s ignorance towards SELinux or any other security implementations over the last couple of years.)  Although iptables is available, per default it’s disabled. But instead they have the cool “ufw” tool, a front-end to the netfilter firewall, as they call it. What it does is using OpenBSD’s pf syntax to create rules for iptables. I guess I should like that, because pf’s syntax makes a lot more sense than iptable’s. Unfortunately, I don’t like any “front-ends” messing with my settings. On Linux, I expect to use iptables as the common standard. OpenBSD’s pf (packet filter) can be found on OpenBSD, NetBSD and FreeBSD. So again, I have to remove unnecessary stuff.

Furthermore, in the enterprise section, I would expect thoroughly tested support for DRBD, GFS2, heartbeat, haproxy et al in order to build solid clusters. However, GFS2 is marked experimental in Ubuntu 10.04. So it has not been tested properly in Ubuntu, which is a shame, because it has been on RedHat Enterprise Linux, where it comes from. And as GFS2 is one of the very few cluster-aware filesystems on Linux, I kind of would expect that to be thoroughly tested (GFS and GFS2 have been out there for years). Or why did Ubuntu Server claim to be an enterprise Linux again? Oh right, must have something to do with the Gentlemen’s agreement between Amazon and Ubuntu to exclusively ease access to Amazon EC2, a proprietary “cloud” (don’t get me started on this term). So what Ubuntu users get is an increasingly strong mix of GPL stuff with proprietary extensions.

Again, I’m not saying that Ubuntu is bad. And I really do appreciate Mark’s effort to create a very good desktop Linux, which keeps up with recent technology development and hardware support. For the server, on the other hand, I am a bit more conservative. I don’t need half-baked support for quite literally everything there. Nor do I need the very latest libraries and features. What I do need is robustness. I prefer a minimal base installation (which includes standard tools and security measurements) and to take it from there. And I prefer things which have been really thoroughly tested. Experimental is a word I don’t really want to read there. Bottom line is that Ubuntu Server feels a bit like an experimental server Linux for beginners.

That, my friends, is why I’m not fond of Ubuntu. Admittedly, I got a bit carried away here. And I do know that many of you (especially the Ubuntu “fanboi” folks ) will disagree. At the end of the day, every systems administrator has got their own preferences. Each to their own. No Ubuntu for me (unless I’m being forced to).

Now bring on the stones you want to throw at me…

First of all, questions like this, which do not tell anything about the author’s aims and intentions, are not answerable. One could as well ask: Ferrari or Landrover? I’d suggest taking the Ferrari for the next cross-country rally, whereas the Landrover is definitely the best choice for the F1 track. Anyway, you got my point.

This blog post has potential for flamewars between the lovers of BSD and Linux, and also between lovers of either of the Linux distributions. So let me emphasize that this is my personal opinion.

Let me kick off with two certainly arguable statements and take it from there:

• Production environment: The operating system of your choice should be the one, which you are most comfortable administering, because it’s your job to secure it to the best of your knowledge and solve upcoming issues within the least possible amount of time and effort.
• Experimental environment: Do whatever you want to. Experimental environments are meant to gain more knowledge, experience or compare it with other environments.

In this context, let’s be clear about this: Any server that is accessible from any other untrusted machine (aka Internet), is a production environment! Why so? Because it could easily be turned into a threat to others (if not secured properly), which can cause trouble with your ISP or with third parties, which leads to costs, and in the worst case lawsuits! This means: Although you run it for your own pleasure, you have to ensure that your pleasure does not become a nuisance to others — be it by your mistake or by third parties taking over your server. Should be common sense, but apparently it’s not.

Now that we’ve understood that the playground approach is misplaced in a server environment, you may want agree with my previous statements.

“I hear you, but which Linux/BSD/Unix is the best for which aims?” Let me first briefly explain how things have evolved and why a FreeBSD user will have problems recommending any Linux distribution.

Unlike any Linux distribution, which strictly speaking is merely the kernel bundled with a bunch of (mostly) GNU tools and programs, FreeBSD is a real operating system, where all core elements are maintained by a central “authority”, the FreeBSD Project (which is funded by donations collected by the FreeBSD Foundation). That ensures a high level of integrity and as a result stability. FreeBSD (like NetBSD) is a fork of the original BSD by the Berkeley University, which was derived from AT&T Unix. Nowadays you find three major BSDs out there: FreeBSD, NetBSD, and OpenBSD (which was forked from NetBSD). They are maintained by their core teams, and cross-port various functionality whenever suitable (e.g. OpenBSD’s packet filter pf). When you install any of these BSD’s base, you will end up with a working operating system and all core tools needed to administer it.

When you install any of the approximately 250 different Linux distributions out there, you more precisely install a third-party bootloader, the Linux kernel, and a whole bunch of third-party (GNU) tools and software. What exactly you end up with, depends on the taste and policies of the distributors. It should be easy to understand that a distribution which focuses on including the latest drivers and software in every release, cannot be as stable as a distribution with a rather long release cycle that has got a big number of enterprise-level users. Essentially they are all the same, but the collection of software and tools (and their branding and look&feel) differs. As various GNU projects have got a lot of cross-dependencies (e.g. PHP with GD, ImageMagick, MySQL, to mention a popular one), it is a tedious and time-consuming task to bundle the right versions with each other in order to get a stable system.

Or in other words: A Linux distributor has to ensure that their selection of third-party software form a stable system, whereas the major BSD derivates maintain the core system themselves. In the BSD world, third-party software isn’t part of the core functionality. Hence BSD doesn’t depend on the good will of other software projects. However, you can of course get a lot of third-party software, too: The portstree (in FreeBSD for example), currently contains over 20,000 different programs, carefully selected and tested, and linked against other ports and/or the core libraries. As the latter are provided by the BSD maintainers, you can be sure to have a solid foundation.

If you look at SELinux, it was a rather chaotic uncoordinated situation in the beginning: SELinux was developed and maintained by the NSA, and was not part of the kernel initially, but you could compile it as a kernel module (don’t get me started on kernel modules on a server). The tools to actually use it are part of the GNU coreutils package — third party software, strictly speaking. When SELinux reached a stable status and was supported by the Linux kernel, some distributors decided to include and activate it by default (Fedora, RHEL, and CentOS), while others didn’t make use of it at all (Debian, Ubuntu). So security was a matter of the distributor’s taste. That happens when there’s no central “authority” which ensures continuity, and coordinates kernel (and related) development. A sad result was that people didn’t want to get used to SELinux, because it wasn’t (and still isn’t) accepted as a standard and must-have. Even nowadays you read recommendations like “use ’setenforce 0′”, which effectively switches SELinux restrictions and its security improvements off! As far as I know, only RHEL and CentOS install and activate SELinux and its utilities by default. They are also the only mainstream Linux distributions which activate the iptables firewall by default, and apply a restrictive ruleset, by the way.

You’ll still even find Linux distributions, which allegedly target the server market, without SELinux utilities installed. How can you ignore huge security enhancements in a server environment? Ah right, the distributor has got a different taste and would probably add no other security tools.

Apologies for my sarcasm. Linux is not all bad, but you must not expect any distribution to be as rock-solid as any of the three main BSDs. Let’s check out which Linux is the least of all evil

The first commercial distribution back in the early 90’s was Slackware, which nowadays is only being used on a minority of Linux-based servers. Slackware is sort of considered geeky.

A couple of years ago, the big players were RedHat Linux in the English-speaking countries, and SuSE in the German-speaking areas. That has changed. RedHat Linux for the commodity market does not exist any more (it is now the community-maintained Fedora Linux, supported by RedHat). RedHat’s own Linux distribution is RedHat Enterprise Linux, which obviously targets enterprise-level customers, who are willing to pay for licenses and professional support. For those who don’t, CentOS as a de-branded RHEL copy has become more and more popular. It claims 100% binary compatibility with RHEL, without asking for license fees, and without offering professional paid support. The target group for both RHEL and CentOS are enterprises and server installations, whereas Fedora targets the desktop market.

Especially in Germany, Debian Linux is also widely used. The easy package management with apt-get certainly played an important role in its success. Fedora introduced yum to make RPM package management as easy. (Open)SuSE seems to lose market share. YaST as their package manager could be one reason.

Ubuntu was started as a Debian derivate just a few years ago, and initially aimed the desktop market trying to keep up with current hardware drivers and new features. Apart from its LTS (long term support) versions, which are being released once every two years, the life cycle of the half-year releases is very short. However, Ubuntu has experienced great success and played an important role in making Linux a widely accepted desktop operating system. In my opinion it is not the best choice for server installations where robustness is more important than introducing the latest features and device drivers, though. It also lacks SELinux utilities in its default installation.

For the tough cookies, there’s also Gentoo Linux, which covers kind of a niche market: people who believe in stability by compiling everything from scratch (and most obviously took BSD as an example), but who are reluctant to leave the Linux terrain towards BSD. Why am I saying this? Simply because compiling from scratch is rather not suitable for newbies, but all the effort still doesn’t provide a BSD level of stability in a Linux environment as too many bits and pieces are actually third-party software.

Personally, although I used to prefer Debian (before SELinux became de facto standard), I cannot take that distribution seriously any more. They made a terrible mistake when they “patched” the OpenSSL library, turning all generated keys and certificates built with them into garbage (or what do we call keys and certs which are created with a predictable random generator?). In my opinion, it shows pretty well why a more centralised approach of maintaining core components is better. There’s a thin line between diversity and mess. When distributors start patching core components just like that, rather than contributing code to the upstream projects, the diversity will soon equal mess — and introduce absolutely unnecessary distribution-related security flaws.

In my personal opinion, CentOS is the Linux distribution for a server setup (or RHEL for those who rely on professional support), whereas a desktop or laptop user’s best bet would be Ubuntu (if you can live with a short release cycle and are happy to update your whole system often) or Fedora.

However, I do prefer FreeBSD for servers (as you could easily tell after reading all this).  And on a desktop/laptop, MacOS is my favourite. (I know that paying for solid software is political incorrect nowadays, but at least it has a reliable — FreeBSD/Darwin — foundation!)