Sysconfig's Blog

Weird title, right? Well, it’s also a weird project, which I am doing just for fun and as a kind of proof-of-concept. The idea is to build a “micro data centre” on a single VMware ESXi based machine and to fully replicate all parts of it onto another single ESXi based machine, which is located in another real data centre. That sounds easy, right? But hold on a second.

Here’s a list of features I want to implement:

• full failover
• high security
• scalability
• load balancing
• prioritisation based on the distance to the micro data centres
• clustered filesystem across micro data centres

The idea is to put a couple of virtual machines on both of the “micro data centre boxes:”

• pfSense firewall
• Pound load-balancer
• MySQL-Cluster nodes (NDB Cluster)
• GlusterFS cluster storage nodes
• PowerDNS with geo plugin (to deliver IP resolutions depending on the source of a query)
• Apache instance(s) accessing the GlusterFS storage and the MySQL cluster

The only VM which has a public IP is the pfSense firewall. It deals with the requests and does the inter-box traffic shaping to ensure stability for the cluster filesystem (which replicates itself between the “micro data centres” in real-time) and the MySQL cluster, which is also spread over both physical boxes. The PowerDNS server resolves domain queries depending on the source of the requests. One server resides in the US, one is located in the UK (for now); so European customers will always be redirected to the UK server, whereas American customers will always see the US based server first.

The reason, why everything lives in its own VM is that choosing this approach, it is possible to easily move a VM onto a more powerful machine if necessary. The whole “micro data centre” can grow as required. But from the beginning — that is the key interest of this experiment — everything is fully redundant and it does not matter, if a single VM or one of the two boxes entirely fails. That’s much closer to 100% uptime than any single-data-centre approach (which you will most commonly find).

Actually, I have never had a look behind the curtains of a Content Delivery Network (aka CDN). But I am pretty sure that the idea is similar, although they certainly do not use VMs or only two servers. So the implementation will differ. Anyway, I’d like to see how my approach works. It’s an experiment using free software only (which includes ESXi since it was released with a new license end of July).

I started this, because I like VMware ESXi and want to stress-test it a bit. Also I like the pfSense firewall (which is FreeBSD based). And I do have good experience with GlusterFS (also not yet in an inter-continental setup). MySQL NDB is new to me. So is location-based domain resolution based on PowerDNS.

I’m pretty sure that VMware Infrastructure with the HA-AddOn would be much easier (and way more expensive!), but this is about learning new things and combining them with existing knowledge… Yes, call me a freak.

So, you now know the idea. In the next articles of this series, I will go into detail about all the things I do and how I am getting on with the project. Stay tuned.

[Added August 17th] Here’s a small diagram how I thing it could look like: