Let’s say you are running FreeBSD on a server, and you need to do a major upgrade (that is from 6.x to 7.x). This process can take ages, if your machine is not running the latest hardware, and/or you have a lot of 3rd party software installed (ports). I’m not talking about an impatient person’s definition of ages, or about the one of a customer, who claims hundreds of quid financial loss in 20 minutes downtime on Sunday morning 1:30 am.  I’m talking about ages as in many hours.

Of course, a FreeBSD upgrade doesn’t require to be offline while it’s proceeding. But you will need to reboot. And as a rule of thumb, one can assume that dependencies in the ports will break. Usually only one or two of them, but it requires manual work, and can cause an unpredictable partial downtime, which is longer than it takes to reboot the machine.

So how can virtualisation help here? In a nutshell, it allows you to do the whole upgrade on another virtual machine. You can take a snapshot of the production machine, start it as a new VM, and do your work there, while the original VM stays online.

This also reduces stress enormously, because if you break something during the upgrade, there’s no time pressure to fix it. You can spend as much time as it takes to finish your work properly. Cool, isn’t it?

And when you’ve finished your work, you can inform your customer about an upcoming 1 or 2 minutes downtime for a major system upgrade (which you have already finished).

All you need to do when the time has come, is to sync files which changed during run-time (for example mail folders), change the network settings in order to make your upgraded snapshot take over, and then you can safely decommission the old VM. It really is as easy as that.

To cut a long story short:

• massive speed improvement
• excellent for work with many small files (compile times for big projects reduced significantly, to give one example)
• good value for money (~ £125)

And now the down-sides:

• getting very hot

When I moved in a year ago, there wasn’t a broadband connection in this house, but there was a phone line — with BT apparently. So I went for the option, which I hoped would be the quickest (in terms of getting connected): BT. 8meg sounded ok to me, and I didn’t want to use 3G all the time (although I had a lot of traffic allowance on my contract).

We were in fact online after a couple of days. However, we did not get 8meg. We were around 6meg first, with a rather ridiculous upstream speed:

But, as we all know BT, things never get better. They only get worse… This year we had “impressive” results, especially in the evenings:

This picture might look like an exception, but it’s not. Towards the end of the contract things got even worse:

I’d like to point out again, that this connection was supposed to be a 8meg connection!

Even my T-Mobile 3G connection could compete with that:

We were told that our line quality is too poor to offer higher speeds. And the exchange was too far away, they said. In fact it’s only about 300m down the street! Plus, one might wonder why we got 6meg downstream half a year earlier then? Why did it constantly get worse, especially in the evenings? Right, because the uplinks from the exchange onwards are blocked. They connect more customers than they actually can handle. Should introduce Congestion Charge there, seriously! So, probably it doesn’t surprise anyone that we did not stay with BT.

Virgin Media’s fibre optic offers were very appealing. Our neighbours next door have it. Our neighbours on the other side next door can have it. And even in the same house, our neighbours upstairs can have it. That’s what the online availability checker said and still says. Guess what? Right, we cannot have it!

I thought a support call might actually help discover an error in their database, and in fact we can have it, too. Well, I thought. Obviously every man-hour is way more expensive than the monthly fibre optic rental customers are being charged. So they showed little to no effort, and told me that I can’t have it, because that would be technically impossible. Why? The cable is out there, and my neighbours upstairs can have it. Why would it be technically impossible? The sophisticated answer was: I don’t know. The system shows it’s impossible and wont change in due course.

A second call, hoping to talk to someone who’s a little bit brighter, didn’t make a difference. Other words, same meaning.

People recommended Be* and reckoned that they had a good customer support, too. Interestingly they estimated a possible line speed of 19meg!  Wow! To be fair, I would have been happy with anything beyond 4meg, which wouldn’t drop in the evenings.

Only one week after we decided to switch, we were connected with Be*. There was only a 3 or 4 hours gap on the day they switched the line. While I was working from home, I could fill that gap with T-Mobile 3G.

One of the first things I did after we got connected was, of course, to verify the line speed with Be*. Here’s the first result:

Yay! But…wait. Didn’t they say 19meg downstream and 2.5meg upstream? So we only got half of the promised speed. Hmm. Of course, still way better than BT ever was — on the same physical phone line! However, we are humans… we always want more, especially when someone promises to give more. So I quickly changed my mind and decided not to be happy to have 4meg or more. That’s probably because I somehow expected that the line wasn’t capable of offering more than the 6meg we initially got from BT. But as it was, I became greedy

After a couple of support tickets and two weeks later, the best results we ever got (relatively stable) were these:

Be*’s support turned out to be good with simple things, and total failure with difficult things. Arguable if support is the right term then. Actually their user forums were more helpful than their paid staff. Ping here, traceroute there, connect via Ethernet rather than wireless… Also they didn’t really read the whole trail of the ticket. Just the last message. Which, obviously, resulted in the same questions being asked over and over again — and a solution being severely delayed.

The funny thing was that the BeBox (the provided DSL router) synced with different speeds on every single re-connect. Plus, it started to reboot randomly, dropping the DSL connection for 5 minutes each. Sometimes 4-5 times a day (probably more often, but we didn’t notice). How would a ping or traceroute help to solve this? (Yes, I have been asked a couple of times to provide pings and traceroutes to google.com and the BBC!)

Anyway, let’s continue…

Be* offers three different DSL profiles (sync setups), which customers can choose from:

• optimised for speed
• normal
• optimised for reliability

Plus, you can opt in for the fast path option, which gives way faster round trip times at the cost of reduced error correction.

The BeBox was using the “normal” profile without fast path. We did try the “optimised for speed” option, but that made things worse. The disconnects really bothered me, but I didn’t want to reduce the speed further. So the picture above shows the maximum we could get through the line using a BeBox, and accepting a couple of disconnections a day. I wasn’t impressed.

In different forums I found people complaining about the BeBox, describing similar symptoms. So I thought: Let’s just try another router, for whatever it’s worth.

I ordered the D-Link DSL-2640B (£50). When it arrived, it took me about 30 minutes to get it running (20 minutes to get the DHCP lease renewed, 10 minutes configuration and reboot).  And here are the very first results (still with Be*’s “normal” profile, and without fast path):

Did you notice? More than 3meg more downstream with exactly the same line and cabling, just a replaced router. And this was an average speed test result. (By the way: I did not only use speedtest.net with their Maidenhead server; I double-checked the results with other tests, too — all of the reported results in this article)

But it gets even more interesting: As the D-Link apparently can deal way better with a below-average line quality, I wondered if I could push it a bit. Today I asked Be* to switch to “optimised for speed” and activate fast path on my line. And now watch this:

To digest this article… On the same physical line, I got:

• 6meg initally with BT
• dropping down to 2meg with BT
• 11meg with Be* and their BeBox
• 16.5meg with Be*, optimised settings and a D-Link router

Using Be* with a third-party router can result in 10.5meg more bandwidth than BT said would be possible on that phone line, and even 5.5meg (50%!) more than Be*’s BeBox can achieve on that line.

The D-Link router didn’t show any uncorrectable errors so far. No disconnects. No other unexpected problems. But very good performance!

Why the heck do the ISPs bundle crap hardware with their offers, causing unnecessary support inquiries? And why do the ISPs not have support staff in place, who are actually capable of dealing with the increased support load then?

Or in other words: Why does the customer have to spend lots of time and a bit of extra money to figure out and solve the issues on their own, while all they requested was to get what they actually pay for?

[ Update: I just realised that Be* switched the line back to the normal profile without fast path active. Sync speed and throughput went immediately back down to the old values. Let's see how long it takes to get my preferred settings activated (and hopefully persisted!) again. I'm getting slightly mad at them. And the option on their "website" (quotation marks on purpose -- see youself!), where I could in theory choose the setting myself, is broken, too. So I have to wait for their support to do it... ]

[ Update 2: About one hour later, I'm back to the desired settings... Let's see for how long. They claim it has been changed on the member portal. I wish I could do that, but firstly it's broken there, and secondly I haven't touched it at all... Funny people. ]

This tiny manual comes without any warranty whatsoever! I only collected information from other sources, which seem to work for me, but not necessarily for anyone else. In fact you will even lose Apple’s warranty with a jailbroken phone. So use it at your own risk.

Here’s how it worked for me without any problems using MacOS:

Needless to say that you must take a backup with iTunes first!

Then, you need to download a firmware for OS 3.0 (not 3.0.1, even if you are currently running 3.0.1), parts of which will later be patched and then uploaded by the jailbreaking application. As 3.0.1 only adds SMS-related security fixes (they won’t be removed), this does apparently work, whereas using a 3.0.1 firmware does not work.

So, open a terminal window and download the firmware file  (not via Safari, because it unpacks the firmware bundle automatically, which then cannot be detected by the application):

curl -o iPhone2,1_3.0_7A341_Restore.ipsw
"http://appldnld.apple.com.edgesuite.net/content.info.apple.com/
     iPhone/061-6582.20090617.LlI87/iPhone2,1_3.0_7A341_Restore.ipsw"

(That goes all into one line, in particular the URL. Ran into display problems with this blog, sorry)

Secondly, you need the redsn0w torrent. If you haven’t got a bittorrent client yet, get the original here.

After you’ve downloaded redsn0w, simply run the application, and point it (“browse”) to the previously downloaded firmware file. The rest is absolutely self-explanatory. 3-5 Minutes later, your iPhone 3GS will be jailbroken, and you will be able to use the Cydia and Icy repositories with all their great (but not Apple-approved) iPhone applications.

When you open iTunes next time, you can easily verify that it still shows Software Version: 3.0.1

Have fun — at your own risk!

References:

• http://discuss.gdgt.com/apple/iphone/3gs/tips/iphone-3gs-background-apps-with-jailbreak/
• http://blog.iphone-dev.org/post/126908912/redsn0w-in-june

Read here, what recently happened to many Twitter employees, including those dealing with confidential documents, which now have been published on Techcrunch.

The English translation of the original source can be found here.

These are the two lessons to learn:

• Do not use third-party services to store confidential information!
• Use passwords and security questions, which cannot be guessed easily!

A password must:

• not be shorter than 8 characters
• not contain only letters (better mix with numbers and special characters!)
• not contain natural language (i.e. words which can be found in dictionaries)
• not contain names, birth or anniversary dates, parts of (previous) home addresses, your favourite colour or hobby
• not be re-used on a whole bunch of different web sites
• not be stored in your email inbox (if a bloody stupid provider sends you non-temporary cleartext passwords, delete them instantly from any online media or computer, and change the password, unless you want the next worm or trojan to forward them to criminal parties)

Don’t think password hacking happens to the big players only. Those of you who have been running their own (web) servers for a while, shall have a look into the auth.log and access.log files (for a start). Hopefully that opens your eyes: Automated password cracking and site hacking attempts are no exception. They happen regularly to all of us. And they happen to all third-party services you use, but there you have no influence whatsoever, hence cannot do anything except making your passwords and security questions as difficult to guess as possible!

Please help making people aware of the necessity of strong passwords. Just share this post via Twitter, facebook, or whatever social network you are member of. Thank you!

First of all, questions like this, which do not tell anything about the author’s aims and intentions, are not answerable. One could as well ask: Ferrari or Landrover? I’d suggest taking the Ferrari for the next cross-country rally, whereas the Landrover is definitely the best choice for the F1 track. Anyway, you got my point.

This blog post has potential for flamewars between the lovers of BSD and Linux, and also between lovers of either of the Linux distributions. So let me emphasize that this is my personal opinion.

Let me kick off with two certainly arguable statements and take it from there:

• Production environment: The operating system of your choice should be the one, which you are most comfortable administering, because it’s your job to secure it to the best of your knowledge and solve upcoming issues within the least possible amount of time and effort.
• Experimental environment: Do whatever you want to. Experimental environments are meant to gain more knowledge, experience or compare it with other environments.

In this context, let’s be clear about this: Any server that is accessible from any other untrusted machine (aka Internet), is a production environment! Why so? Because it could easily be turned into a threat to others (if not secured properly), which can cause trouble with your ISP or with third parties, which leads to costs, and in the worst case lawsuits! This means: Although you run it for your own pleasure, you have to ensure that your pleasure does not become a nuisance to others — be it by your mistake or by third parties taking over your server. Should be common sense, but apparently it’s not.

Now that we’ve understood that the playground approach is misplaced in a server environment, you may want agree with my previous statements.

“I hear you, but which Linux/BSD/Unix is the best for which aims?” Let me first briefly explain how things have evolved and why a FreeBSD user will have problems recommending any Linux distribution.

Unlike any Linux distribution, which strictly speaking is merely the kernel bundled with a bunch of (mostly) GNU tools and programs, FreeBSD is a real operating system, where all core elements are maintained by a central “authority”, the FreeBSD Project (which is funded by donations collected by the FreeBSD Foundation). That ensures a high level of integrity and as a result stability. FreeBSD (like NetBSD) is a fork of the original BSD by the Berkeley University, which was derived from AT&T Unix. Nowadays you find three major BSDs out there: FreeBSD, NetBSD, and OpenBSD (which was forked from NetBSD). They are maintained by their core teams, and cross-port various functionality whenever suitable (e.g. OpenBSD’s packet filter pf). When you install any of these BSD’s base, you will end up with a working operating system and all core tools needed to administer it.

When you install any of the approximately 250 different Linux distributions out there, you more precisely install a third-party bootloader, the Linux kernel, and a whole bunch of third-party (GNU) tools and software. What exactly you end up with, depends on the taste and policies of the distributors. It should be easy to understand that a distribution which focuses on including the latest drivers and software in every release, cannot be as stable as a distribution with a rather long release cycle that has got a big number of enterprise-level users. Essentially they are all the same, but the collection of software and tools (and their branding and look&feel) differs. As various GNU projects have got a lot of cross-dependencies (e.g. PHP with GD, ImageMagick, MySQL, to mention a popular one), it is a tedious and time-consuming task to bundle the right versions with each other in order to get a stable system.

Or in other words: A Linux distributor has to ensure that their selection of third-party software form a stable system, whereas the major BSD derivates maintain the core system themselves. In the BSD world, third-party software isn’t part of the core functionality. Hence BSD doesn’t depend on the good will of other software projects. However, you can of course get a lot of third-party software, too: The portstree (in FreeBSD for example), currently contains over 20,000 different programs, carefully selected and tested, and linked against other ports and/or the core libraries. As the latter are provided by the BSD maintainers, you can be sure to have a solid foundation.

If you look at SELinux, it was a rather chaotic uncoordinated situation in the beginning: SELinux was developed and maintained by the NSA, and was not part of the kernel initially, but you could compile it as a kernel module (don’t get me started on kernel modules on a server). The tools to actually use it are part of the GNU coreutils package — third party software, strictly speaking. When SELinux reached a stable status and was supported by the Linux kernel, some distributors decided to include and activate it by default (Fedora, RHEL, and CentOS), while others didn’t make use of it at all (Debian, Ubuntu). So security was a matter of the distributor’s taste. That happens when there’s no central “authority” which ensures continuity, and coordinates kernel (and related) development. A sad result was that people didn’t want to get used to SELinux, because it wasn’t (and still isn’t) accepted as a standard and must-have. Even nowadays you read recommendations like “use ’setenforce 0′”, which effectively switches SELinux restrictions and its security improvements off! As far as I know, only RHEL and CentOS install and activate SELinux and its utilities by default. They are also the only mainstream Linux distributions which activate the iptables firewall by default, and apply a restrictive ruleset, by the way.

You’ll still even find Linux distributions, which allegedly target the server market, without SELinux utilities installed. How can you ignore huge security enhancements in a server environment? Ah right, the distributor has got a different taste and would probably add no other security tools.

Apologies for my sarcasm. Linux is not all bad, but you must not expect any distribution to be as rock-solid as any of the three main BSDs. Let’s check out which Linux is the least of all evil

The first commercial distribution back in the early 90’s was Slackware, which nowadays is only being used on a minority of Linux-based servers. Slackware is sort of considered geeky.

A couple of years ago, the big players were RedHat Linux in the English-speaking countries, and SuSE in the German-speaking areas. That has changed. RedHat Linux for the commodity market does not exist any more (it is now the community-maintained Fedora Linux, supported by RedHat). RedHat’s own Linux distribution is RedHat Enterprise Linux, which obviously targets enterprise-level customers, who are willing to pay for licenses and professional support. For those who don’t, CentOS as a de-branded RHEL copy has become more and more popular. It claims 100% binary compatibility with RHEL, without asking for license fees, and without offering professional paid support. The target group for both RHEL and CentOS are enterprises and server installations, whereas Fedora targets the desktop market.

Especially in Germany, Debian Linux is also widely used. The easy package management with apt-get certainly played an important role in its success. Fedora introduced yum to make RPM package management as easy. (Open)SuSE seems to lose market share. YaST as their package manager could be one reason.

Ubuntu was started as a Debian derivate just a few years ago, and initially aimed the desktop market trying to keep up with current hardware drivers and new features. Apart from its LTS (long term support) versions, which are being released once every two years, the life cycle of the half-year releases is very short. However, Ubuntu has experienced great success and played an important role in making Linux a widely accepted desktop operating system. In my opinion it is not the best choice for server installations where robustness is more important than introducing the latest features and device drivers, though. It also lacks SELinux utilities in its default installation.

For the tough cookies, there’s also Gentoo Linux, which covers kind of a niche market: people who believe in stability by compiling everything from scratch (and most obviously took BSD as an example), but who are reluctant to leave the Linux terrain towards BSD. Why am I saying this? Simply because compiling from scratch is rather not suitable for newbies, but all the effort still doesn’t provide a BSD level of stability in a Linux environment as too many bits and pieces are actually third-party software.

Personally, although I used to prefer Debian (before SELinux became de facto standard), I cannot take that distribution seriously any more. They made a terrible mistake when they “patched” the OpenSSL library, turning all generated keys and certificates built with them into garbage (or what do we call keys and certs which are created with a predictable random generator?). In my opinion, it shows pretty well why a more centralised approach of maintaining core components is better. There’s a thin line between diversity and mess. When distributors start patching core components just like that, rather than contributing code to the upstream projects, the diversity will soon equal mess — and introduce absolutely unnecessary distribution-related security flaws.

In my personal opinion, CentOS is the Linux distribution for a server setup (or RHEL for those who rely on professional support), whereas a desktop or laptop user’s best bet would be Ubuntu (if you can live with a short release cycle and are happy to update your whole system often) or Fedora.

However, I do prefer FreeBSD for servers (as you could easily tell after reading all this).  And on a desktop/laptop, MacOS is my favourite. (I know that paying for solid software is political incorrect nowadays, but at least it has a reliable — FreeBSD/Darwin — foundation!)

First we need to look at what Adobe Air is: It is a run-time environment, ported to all common operating systems, which allows to run applications built with Adobe Flex on almost any desktop rather than embedded with a browser (where you can run the same applications with little or no modifications using the well-known Flash Player Plugin).

So, we know that AIR developers use Flex. Why is that? That’s as easy to answer: Flex targets front-end/GUI designers in the first place. Together with Flex, Actionscript, and Flash, you can build quite neat and shiny user interfaces. Most of the “business logic” (I know that sounds funny in this context) is not part of it though. The strength of Flex is to use external data sources like web services. On the one hand, it’s certainly a limitation, but on the other hand that’s exactly why it’s being used for Twitter clients, which don’t do much more than using Twitter’s API (web service) to display tweets, subsets of tweets, search results, friends and stalkers (erm followers, of course).

So if you want to bring a desktop application to the market, which heavily depends on web services, Flex is probably one of the fastest ways to get there. And Adobe AIR brings the application to the desktop — on all common operating systems.

Yes, it’s proprietary software, and closed source. However, anyone can use it free of charge (the Flex SDK is free, the not required Adobe Flex Builder is not). Pretty similar to PDF and Flash. But the licensing policy is another discussion, anyway.

Ok, everyone has Facebook (at least in English speaking countries people would rather ask if you are on Facebook than what your phone number is). And then there’s Twitter, another way of keeping friends or customers or whoever informed about what’s going on. Plus, you need to share you favourite URLs with digg, del.icio.us, or any other social bookmarking service of your choice. And of course, you need your own blog! And a Flickr account for your photos. Did I cover all of the services a modern Web 2.0 person has to have? Probably not. But now the core question: How do you manage to keep everything up 2 date? I mean as someone who’s already got a full-time job…

Comments, suggestions, URL appreciated!

I have to admit that I really like C#. The last 4 or 5 months, I’ve hardly touched any other programming language. And so it happened to be my first choice when I was asked to develop a “reasonably fast” logging server.

So what is this Octopus thing about? In a nutshell it’s a server, which takes HTTP requests with a defined set of URL parameters, confirms receipt the client, and then asynchronously stores the request into a database. The interesting bit is the part between accepting the request and storing it to the DB. 

Basically, we’ve got three levels of storage: a in-memory queue, a flat file buffer, and the database. The tricky thing was to optimise processing, while still being fault-tolerant. I kind of managed that, I think. Octopus can (depending on the hardware it runs on) easily accept up to 2,000 requests per second, and will store them into either a MySQL or SQLite database at the moment. Please read more about it on the Octopus Website. ]]> http://sysconfig.ossafe.org/2009/06/octopus-logging-server-in-c/feed/ 0 http://sysconfig.ossafe.org/2008/09/waking-up-from-hibernation/ http://sysconfig.ossafe.org/2008/09/waking-up-from-hibernation/#comments Thu, 25 Sep 2008 23:01:41 +0000 admin http://blog.the-ally.co.uk/?p=26 After years of Systems Administration I have completely forgotten how much I used to like developing in Java. My current employer gave me that opportunity recently. So I took ownership and responsibility to de-hibernate myself (my Java knowledge became a bit rusty) and to start working on a project which separates the website’s business logic layer from the database layer. That becomes necessary as we want to re-design our infrastructure in a way that enables us to be as platform independent and as scalable as possible.  

On the one hand, we are not even entirely sure if we’ll stick to MySQL or switch to any other DB server type, hence we must not use SQL dialect specific code within the application. On the other hand, we want to provide a SOAP endpoint internally, which allows us to access data from all sorts of applications and clearly separates the business logic of all sorts of clients from our actual data store.

In addition, caching of object structures which are mapped against data structures (or tables) would be very neat. Consequently, I suggested and started using Java with Hibernate and JAX-WS on a Glassfish Java Application Server. We might integrate an additional caching layer later. Terracotta is one of the candidates, which integrates seamlessly and can increase speed by factor 2 to 10. 

After one week of researching, developing proofs of concept and presenting a first prototype, I am sooo excited about all that.  I always loved Java, and again I remember why.