Building My Lab – Physical, Virtual, and Imaginary

Since I am not a full time sysadmin any more (or really even a part time sysadmin…can you be a “hobbyist” sysadmin?), I’ve lost one of my most valuable resources for learning new things…a decent lab.

I guess I never really understood how important it was to have a pile of legacy kit sitting around waiting on me to hook it together (or, more likely, tell my junior admin to hook it together for when I had time to play). When I missed my whiteboard, I went to Home Depot and got a bunch of dry erase panel and covered the garage in it. My wife thought that was weird, but she basically understood. I’m not sure she’d be so tolerant if I asked for a rack and some servers to fill it.

For this reason, I’ve been investigating what my options are. There are some easy ones, of course. I’ve got a newish unibody MacBook Pro, so VirtualBox runs great on it, especially with 8 GB of RAM. You can technically upgrade it to 16GB, but that’s really expensive, so I’m not going to do that just yet. Besides, at this point, my laptop is very I/O bound.

Incidentally, you CAN run ESXi(vSphere) under VirtualBox, but it’s pretty much a “cause I want to practice with the interface”-only kind of thing. You can run VMs, but it’s not really pretty.

So although I can do some simple prototyping in VirtualBox, in order to do some heavier lifting, I’ve got to figure something else out. I actually have two pieces of physical server hardware that I picked up from a friend. One is an old Dell machine that is technically 64 bit, but was made before the VT-x extensions. The other is a 32bit HP Proliant ML570 7u beast. It’s got some possibilities in terms of disk space, since there are two 6-disk arrays in it, but it’s huge and loud and expensive in terms of electricity usage. (By the way, if you’re interested in either of these, just say the word and name your price).

Honestly, I’ve been thinking about picking up a couple of these PowerEdge 1950s. They’re decently upgradable, they’re 64 bit with VT-x, they’re on the ESXi HCL, and two of them are still less than half of the space taken up by the HP in my garage.

I’m looking into the cheapest way to get some kind of shared storage solution, which will probably end up being one of the BSD appliances that run ZFS on a bunch of hard disks (something like NexentaStore, maybe). The real question is what I’m going to put the hard disks in. The ML570 is just a bit long in the tooth for this kind of thing, and as I said, really inefficient. Something like this might be nice and unobtrusive.

It's dangerous to go alone. Here, take this cloud security blanket.

One of the things I’ve been doing in the past week or so is to start investigating Amazon’s pay-to-play Amazon Web Services. It took a while to set up the environment on my laptop that I use to administer the machines remotely, and it took a little longer to really kind of figure out the Amazon mindset, but I think I’m actually starting to get the hang of it, and I haven’t even cost myself more than $5 yet.

The thing about Amazon that is kind of scary is that you really are charged per hour. What this means is that I don’t run my Amazon lab for long periods of time. In fact, I’m kind of paranoid about it, so every once in a while, I’ll run ec2-describe-instances just to make sure I’m not accumulating machine charges. I’ve made my instances all of the micro variety, since that’s only $0.03 / hour that the instance exists (note: it’s not enough to shut down the machine; that doesn’t kill the instance. You need to terminate the instance, which also erases the ephemeral storage.

I would be shirking my responsibility if I didn’t also mention that in the spirit of drug dealers everywhere, Amazon is offering a “first one’s free” policy, which gives you a 7500 hours worth of free micro instances. So check that out.

If you have any interest in a “dipping your toes in the AWS water” type post, comment below and let me know. I’m starting to dig the AWS thing.

I haven’t investigated the Rackspace cloud much. The prices seem alright (actually, the tiniest servers on Rackspace are only $0.015 / hour. I’m not sure how much you can do with 256MB of RAM for an hour, but it’s great for testing implementations, which is really all I’m doing.

I’ll probably start playing with the Rackspace stuff eventually, too. There just aren’t a ton of options in this arena to play with. Oracle has a cloud offering, but it all seems to be Oracle as a Service (or OaaS, something I’ve said phonetically for YEARS when talking about Oracle). If someone from HP wanted to invite me into their cloud beta offering, I wouldn’t complain, either.

[EDIT] As it turns out, I got into the HP cloud beta. Sweetness! Thanks HP! I wrote a walkthrough on setting up an HP cloud instance and getting IPv6 up.

Anyway, that’s where I stand. I’m itching to get a more permanent lab environment running here, but in the meantime, I’m relying on the cloud providers to get my technology fix. Anyone have any suggestions or want to talk about the cool stuff they’re doing with their lab so I can be jealous?

  • Matt,

    I use the Dell 1950’s in my lab (currently 3 of them). They’re great for VMware, Hyper-V or whatever you want. I always get them with DRAC cards so I can save myself the walk to the rack to do power ups, or reboots.

    As for AWS, I’ve messed about with AWS, Rackspace, GoGrid, Atlantic.Net, with some online benchmarking on my blog.

    Keep up the good work!


  • I still have some work to do on it, but my home lab has a Shuttle SH67H3 + i2600s + 32GB RAM running ESXi 5 off a 4gb low profile usb flash drive… storage over iscsi/nfs is via a HP Microserver, 8GB ECC RAM, FreeNAS running on another 4gb low profile usb flash drive (internal usb port even).

    I’m going to be putting another ESXi box (desktop i7 quadcore, 24gb ram) across the street in the store, 15ms latency/2mbps usable bidirectional bandwidth, for SRM/Zerto/etc. ON that side I’m likely to run Openfiler with the FC terminator hacks so I can run my SANBOX fc switch over there.

    I have a blog draft waiting for polishing… with details and costs and considerations. But not including hard drives, I think the above comes in around $1k.

  • For the moment I don’t have anything by way of test-lab at home, but when I did I just used a few Shuttle’s. They’re small, powerful (modern processors, lots of RAM), pretty quiet and fairly cheap. They were so quiet I could happily sleep in the same room as them whilst they were running.
    The only negative was lack of RAID options at the time (but hey.. it’s a tiny case, where were the hard disks going to go, exactly?), so it wasn’t necessarily reflective of IO performance you’d get from a full server.

  • Jeremiah Roth

    I just ordered an HP MicroServer for about $250 that I plan to turn into a storage server. I understand it will run ESXi off USB, and then you can install OpenSolaris/Indiana and then use the four spinning disks for your zpool.

    It’s nothing I’d want to use for intensive VMs, but it should be fun to play around with before I make it my iSCSI backup/archive device.

    And yes, I’d love to see a blog post about getting started with AWS.

  • Scott

    “I’ve made my instances all of the micro variety, since that’s only $0.03 / hour that the instance exists (note: it’s not enough to shut down the machine; that doesn’t kill the instance. You need to terminate the instance, which also erases the ephemeral storage.”

    Look into using instances with EBS-backed storage. When you shut those down you stop paying the hourly rate. You don’t have to terminate them. You’ll pay the monthly storage costs of the EBS volume, but that’s something like $0.10 per GB per month, so it’s pretty affordable.

  • Scott

    The setup I just got done prototyping on AWS is a load balancer that sends traffic to 2 micro instances running nginx, which proxy connections to 2 apache servers, which connect to an RDS database instance. I tied the 4 micro-sized websites together with glusterfs, so that when Drupal’s boost module writes out its static cache files on the apache servers, they get mirrored up to the nginx boxes automatically.

    That was pretty fun.

  • To ESXi and their goofy hardware requirements I say “pah” and “feh”! I dropped less than a grand on a Dell T310 with 8GB of RAM and 256GB RAID1 (on a PERC6 no less!). With this mighty beast, I’ve got 10 Xen VMs running happily, serving up webpages, handling emails, giving me plenty of spare capacity to play with. Get a rock-bottom T310 and upgrade the RAM via Kingston and the disks via (with cages from eBay). You can get a whole lot of system without having to constantly stress about “are my Amazon/Slackspace instances suddenly hoovering my wallet for microbucks/minute. My recommendation is to “buy once, cry once”, and get yourself a physical system with some kind of vendor support.

    Don’t get me wrong, I freakin’ LOOOOOVE VMWare (my work has given me a few odd Enterprise Plus clusters to play with, on NetApp and Equallogic SANs to boot), but for home lab setups, it’s hard to go wrong with Xen and/or KVM on a proper physical server.

    Just my $0.02…

  • I’ve recently set up a VPS and moved over my hosted website. It’s a fun place to experiment and relatively cheap, too – if you are a LOPSA member you can get their $12/mo vps for $8/mo.

    The big advantage of this approach is it’s flat fee – no worrying about usage charges accumulating. To be fair though, it’s just a 512MB memory system, which has proven barely enough to run my website with a mysql backend.

    Anyway, just wanted to offer that suggestion if you are looking to set up a ‘virtual lab’.

  • If you are interested in AWS and going to be at the Professional IT community Conference on May 11 – 12 there is a training class on AWS. Check it out at

  • Troy

    Looking forward to your forthcoming AWS blog. I’ve used Rackspace Cloud before so it would be good if you do a comparison.

    Also, have a chat with the dudes at They’ve got their own cloud offering,, which I’m sure they’ll let you take for a spin.

  • Couple of my colleagues use HP Microservers as well. They still do a cash rebate on them I think as well. Throw away the ECC RAM in it, and fill it with cheap and nasty stuff, and you are good to go.

  • Chris: ECC is pretty cheap ($70 for 8GB), and (from what I read) better if you’re doing ZFS. I went with “cheap” 8GB sticks in my Shuttle but decided ECC was worth it for the storage array. And it all fits in my IKEA entertainment center without angering my girlfriend (much). :)

    Phil: prgmr (or any virtual hosting place) might be a bit expensive if he wants to emulate a fully functional environment of any sort… could do a D525 with 4GB of RAM for about $300 which would be ~8 $8 VPSes… break even in 5 months *and* can reuse it as a media center later if he wants to :)

  • Scott

    Robert: even $300 seems pricey when I can run 8 micro instances at Amazon for $0.16 / hour (they just dropped the price to $0.02 / hour today) and about $8 / month for 80GB of storage split among them.

    That’s about 9 months for $300 assuming you run them 40 hours per week, and I run my lab a lot less than that.

  • Jim

    +1 for the AWS post

  • HP Microserver.
    We have 4 of them.
    3 in the lab, one of which acts as shared NFS storage for esx.
    A 4th microserver is off site across a wan and acts as the backup server for the other 3 microservers, plus our general office data.
    Power consumption is nil…about 30 watts.
    A KVM card can be had for about £60, and this give full virtual console, reboot etc
    They can be a bit limiting at times, but since we have had them, the dell 1950s have rarely been turned on.

  • Pingback: Mosh Pit, or Why Terminal Emulators Suck | Standalone Sysadmin()