On the road again...

Date August 27, 2010

My datacenter migration (or renovation, as I'm referring to it) includes a fair amount of added virtualization. We'll be maxing out the memory and processor power of three machines at each site, and those will act as a VMware HA cluster (we're buying the vSphere Essentials Plus license kit for each site).

Of course, I've got to have some VMs to run. I could reinstall all of my machines using cobbler (which would invoke the gods of trial and error, not to mention incur Murphy's Wrath), or I could convert the machines that already exist from physical to virtual (p2v). That second option sounds much less error prone.

That being said, converting a physical machine to a VM isn't exactly a fast process. Hoping to get it done the weekend of the move would be foolish, so I need to get it done beforehand. That's why I'm driving to Philadelphia today.

Last week, I threw a couple of terabyte SATA drives into a spare PowerEdge 1950 server, upped the RAM a bit, and installed a freshly minted copy of vCenter Hypervisor 4.1 (formerly known as ESXi). I'm trucking this machine down to our secondary data site today so that I can begin the p2v conversion process. I've got enough disk space that I won't run out (I'm only putting the root partitions in the VMs, since all the data is stored on the SAN), and I don't need to actually run the machines, so RAM won't be a problem. This will just be a holding tank until I get the VM hosts setup during the conversion weekend.

The actual conversion will be done using VMware Converter, a free tool by VMware that I've been really impressed with. It does want an ESXi...err..vCenter Hypervisor server to connect to, but that's free too.

Once this is down there, I've got some decisions to make. Namely, I need to decide how long to wait until I do the conversion. Not a lot of data changes on the root partition. It's going to be limited to logs, really (since I haven't gotten a centralized syslog server running yet). The exception to this rule is the domain controller at that site. That needs to be the absolutely last machine I convert, and once I do it, I've got to turn off the source, because if the image becomes too far out of sync, well...that's sort of like crossing the streams.

So, has anyone else pre-converted VMs like this in preparation for a move? Any advice or caveats to watch for?

Fixed the mistaken Ghostbusters quote. Did I seriously say "crossing the beams"? I am disappoint.

  • http://jeffhengesbach.blogspot.com/ Jeff Hengesbach

    Good luck Matt. Yeah the P2V process is definitely not speedy by any means and for no apparent reason I've been able to pinpoint.

    About P2V'ing your DC. I've seen where people were able to do it no problem, but I also see many advising not to. If your current DC is a not a 'complicated' configuration, I'd suggest building a new VM, running DC promo, them transferring the FSMOs and such off the physical box(easy process). Personally I've not had a DC P2V go smoothly(ended up doing authoritative, this that and the other things). If you must do the P2V the best advice I can pass on is to do a cold conversion(downtime).

  • Anthony

    For the sake of time it makes sense to P2V in this situation, and it does work very well - usually. Be prepared for gotchas - you are effectively taking the Hard disk out of one machine and sticking it into another with completely different hardware and that can confuse the hell out of some operating systems.

    If you have the time and resources it is probably worthwhile to plan to eventually replace all of the P2V'd systems with fresh clean "installed as a vm" systems and migrate processes and data over - it's also a good chance to review and upgrade/update software and process/policies etc, and check backup strategies.

    If you are running Linux on any of these systems watch out for changes in partitioning - the P2V doesn't know the difference between a Logical partition and a physical partition so if your source machine is using LVM, your VM will have partitions for each LVM - which potentially could blow-up if you have a lot of LVs.

    Also, double check the RAM allocation before you power on the VM's. I'm not sure how P2V decides how much ram to allocate but you'll want to verify it yourself afterwards - I've seen some very strange numbers.

  • http://www.standalone-sysadmin.com Matt Simmons

    You're not the only person suggesting that to me. I'm heavily considering it. I've got to go through the DC config to make sure there's nothing else "fun" going on. Thanks!

    That sounds like a good idea. I've been planning on replacing a lot eventually anyway. As RIPienaar mentioned to me on twitter, it would be a lot easier to replace the servers if they were all running puppet :-) It'll happen, definitely! Thanks for the comment!

  • http://www.cmdln.org/ Nick Anderson

    I've only ever used p2v when I didn't have another option. (ancient machine with outdated software and specific configuration that no one cared to take time to migrate the config but "needed").

    It worked ok, I just don't like the idea, makes the system feel "dirty".

    I had to stand up a new server today when another one died. No restoring from backup. I just installed a minimal vm, then puppet then done. Had some version difference to work out with puppett since these machines have a no touch policy and I wasn't given much time to make sure I had a repo clone when originally setup. But after sorting that out the machine was production ready in about 10 minutes.

  • Andrew from Vancouver

    Matt, VMware Converter does a nice job. The most likely thing you need to fix up is the network card bindings. Stray hardware and software to support it like OEM monitoring and RAID configuration software will be orphaned.

    As Jeff and Anthony said, converting a DC is unreliable, and your best conversion is a cold boot one. The magic phrase you don't want to deal with is "USN rollback". The best practice is to build a new DC and move the FSMO roles to it.

    You can always disable the NIC in ESX before you power on the virtual guest, then use the console to log in and check for "damage".

    Doing a P2V with lag time is not a big deal, you just do a backup before the P2V, then another at go-live time, power down the original server, and do a differential restore to bring up the changes, if any.

    One flukey thing that can break is a domain membership if the machine password just happens to have changed in the meantime. No big. Just join WORKGROUP and then join the domain again. The local groups and imported domain security groups will be unaffected.

  • http://blog.rootmytoaster.org Kenny

    Son I am Dissapoint. As punishment for messing up that quote (which I didn't see you mess up, but I'm going to give you crap for doing it anyway) you must fly out here and watch that movie with me. Maybe hang out, find something cool to do, like drive to Vegas.