Cobbler or just straight kickstart for VMware ESXi?

Date August 9, 2010

I'm working on automating some installs that are going to happen during the infrastructure upgrade, and I need to decide what I want to use for automation.

I have used Kickstart before, and it's essentially a single file that contains instructions for the RedHat installer (although Debian is in on that action, too). The idea with Kickstart is that your "normal" installation (whether that be through DVD, USB key, PXE, or whatever) points to the kickstart file, and the installation proceeds according to those instructions.

Cobbler goes the extra steps and becomes the installation server, PXE/DHCP boot provider, etc etc, in addition to working with kickstart files. In fact, it can even do crazy kickstart templating. It certainly seems full featured, and I've heard people recommend it before.

One of the coolest things I've seen it be able to do is automate new virtual machines. As I understand it, you basically hit a button and a VM is created, powered on, and installed according to the kickstart templates. That's slick.

Unfortunately, the best support for Koan (the Cobbler client) is on Qemu/KVM. The site mentions support for VMware Server, but that's anathema. There doesn't appear to be support for ESXi (certainly not 4.1, which was just released last month), but I was hoping for something more recent than a question on VMware Communities from 2007.

So I come to you. If you've got an ESXi infrastructure, do you automate rollouts? Am I just doing this wrong? I'm leaning toward manually spinning up machines and using Cobbler / Kickstart to perform the installs (maybe with customized boot media, in the case of just kickstart). What do you suggest?

25 Responses to “Cobbler or just straight kickstart for VMware ESXi?”

  1. Jeff Hengesbach said:

    Hi Matt,

    Actually ESXi 4.1 does have Kick Start support. Kendrick Coleman (@KendrickColeman) just last week posted an ESXi 4.1 KS script to his blog page. It is currently work in progress but should be an excellent starting point.

    I'll be building out a couple new 4.1i boxes this week and plan to give it go.

    -Jeff

  2. Jeff Hengesbach said:

    It might be useful if I included the link to Kendrick's article: ESXi 4.1 Kickstart Install - WIP ;)

  3. chewyfruitloop said:

    our vms are generally direct clones of one of the templates I've made
    absolutely no automation at all

    we're supposed to have a pxe image that auto installs win 7, but since we moved DHCP and DNS to this new InfoBlox device the image won't fire up...don't ask it a lot of buck passing

  4. Tweets that mention Cobbler or just straight kickstart for VMware ESXi? | Standalone Sysadmin -- Topsy.com said:

    [...] This post was mentioned on Twitter by Matt Simmons, Planet SysAd. Planet SysAd said: Standalone Sysadmin: Cobbler or just straight kickstart for VMware ESXi? http://bit.ly/cx5pBS [...]

  5. Greg said:

    We've built an automated bare-metal-to-functional-server installer at $JOB. The network installer environment consists of a TFTP/PXE/DHCP/DNS/NFS/HTTP server with various forms of deployment scripts/configs (preseed files for Debian/Ubuntu, Jumpstart for Solaris) which allows a host with blank disks to end up installed with a base set of packages in about 5 minutes or so, with no interaction after selecting your OS.

    VM's or physical doen't really make any difference here; if it can see the right network and is capable of PXE booting (ESXi VM's are), you're good to go.

    The next steps are to automate some personalisation during the install (hostname / IP address), and automatic registration with the Puppet infrastructure to allow for automated provisioning to the point of the new host doing real work.

  6. Greg said:

    @Jeff - I'd be reluctant to use something that directly edits files in Tech Support mode if you're keen on keeping your environment supported.

    As an alternative, look into VMware's vCLI, shipped either standalone or as part of the vSphere Management Assistant (vMA) to either implement or emulate almost everything in that script. By scripting a chain of vCLI commands, you can achieve the same goal without accessing tech support mode.

  7. Lee said:

    I've had GREAT success with SystemImager (http://wiki.systemimager.org/index.php/Main_Page) for exactly this sort of problem.

  8. Farzad FARID said:

    For my current customer who owns a VSphere ESX 4.0 cluster I have setup a Cobbler/Kickstart infrastructure with custom modular kickstarts (everything customizable and selectable with Cobbler classes: RPMs, paritions, standard scripts & configs Oracle 10G/11G deployment, etc.).

    We use this to deploy both physical and virtual servers. Cobbler does not support VSphere, but we just have to create a VM manually, report the MAC address in a new Cobbler "system" and that's it.

    Having to create the VM system manually in Cobbler is just a minor annoyance for me given the huge benefits Cobbler brings: we can deploy complex / ready to use servers in less than 20 minutes.

    Regards
    @Farzy

  9. Matt Simmons said:

    @Jeff

    That looks really promising! Thanks for the link. I'll read through it and see how it works out for me.

    @chewy
    Yuck. I hate black boxes. Good luck figuring out the incantation to make it work :-)

    @Greg
    Nice! I know that whatever solution I put in place will have to be an all-in-wonder like yours, since when I'm going to be installing these, there won't be network services to speak of. I'm not finalized on which solutions I'll be rolling onto the box. I suppose this discussion is part of that process

  10. Farzad FARID said:

    I forgot to say that we're about to integrate kickstart files & scripts for the installation of ESX 4.0 servers themselves in Cobbler.

    But for the moment we have some difficulties to get the kickstart installation to run standalone as some "esxcfg-*" commands do not seem to be executed correctly during the installation..

  11. Matt Simmons said:

    @Lee - Interesting, I hadn't heard of SystemImager. That does sound cool. Thanks for the link

    @Farzad - That sounds pretty much exactly like what I'm trying to do, sans the Oracle stuff. Thanks for the comment. At least I know I'm not headed in completely the wrong direction :-)

  12. Steve Webb said:

    I use cobbler with LSXi at $JOB and we just get a mac address from ESXi and go from there. There's nothing special that we use with ESXi to get cobbler to work friendly with it. Just boot from the network card, if your VLANs are set up properly, cobbler will hear the DHCP broadcast and your machine should connect and do its thing.

  13. Farzad FARID said:

    Oh I forgot another interesting information :) We use Opscode's product, <b<Chef, for configuration management. I fully inegrated Cobbler and Chef by making Cobbler deploy the Chef Client and register the new server in Chef.

    For this I have written a Cobbler extension. This extension also maps Cobbler "classes" into Chef "roles".

    And it works like a charm: as soon as the newly installed server reboots, it connects to the Chef server and updates its configuration according to the roles it belongs to.

    I haven't written a documentation or article on the subject yet but if anybody's interested fell free to contact me and I'll help you.

  14. Stan Chan said:

    Just trying to figure out what the requirements are for the ESXi integration. Can someone give me a general workflow for the use case? I have forked Cobbler/Koan for some of my clients projects to integrate some additional requirements and I would like to add additional support with ESXi/OVF images if it is a trivial effort. I'm hoping to get it merged upstream eventually to get that support in.

  15. Stan Chan said:

    On a side note, at a glance, libvirt is capable of accessing ESX/ESXi drivers using the vSphere API.

  16. Nick Anderson said:

    You are using puppet now right? You could create a small template and then use puppet to do the rest. This has the additional benefit of getting more configuration documented in your configuration management system.

    I still prefer to do initial installs via a kickstart or debian preseed as opposed to cloning a template. But I think it's 6 of 1 and half dozen of another if your only looking at the one rollout. Long term though being able to network install virtual or physical followed by configuration management is best.

  17. Cos said:

    FWIW, I've been doing PXE boot with Ubuntu preseeding to get started, then using Puppet to do the rest for VMs on our new ESXi servers.

  18. Ade said:

    Like others above, we're using Cobbler and Puppet to provision new VMs with no major problems; just make sure your VM is on the same network as the servers initially, or generate an ISO in Cobbler. Oh, and don't configure the VM with a VMXNET3 NIC, as the driver isn't in default RHEL so it won't net boot. You can use VMware's OSP repo as a source to automate VMware Tools package installs if necessary (although our VMware admins don't like this as they prefer to manage the Tools themselves).

    The trickier part is scripting the actual VM creation, which I haven't looked at yet. Could do with some sort of framework wrapped around vCLI.

  19. natxo asenjo said:

    We're doing pxe with FAI and cfengine to get everything working (debian shop here). For testing purposes I tried pxe + kickstart + cfengine for vmware (3.5 still) and it works a treat.

  20. Al Hoang said:

    Cobbler is great if you are going the Redhat route as importing a new version of Centos/Fedora/Redhat is quite simple once you get Cobbler installed and bootstrapped. If you are trying to be more agnostic on (Linux) OS, I would look more closely at SystemImager

    Spending time to get puppet installed and inserted into your provisioning setup will definitely save time configuring the servers as you see fit. If you google around for Cobbler and puppet there are some other people that went this route and it should not be that difficult to follow in their footsteps.

  21. Al Hoang said:

    One more note... I think it would be prudent to spend time simplifying the process where you spin up a new ESXi guest and have it do a PXE boot / installation on guest bringup. Some other people have already mentioned the same approach in the comments. That part will be ESXi specific and can probably be encapsulated in a (not so) simple script that is tailored to your infrastructure needs (usual parameters are memory, disk, and CPUs).

    Once that part is completed, you can use more 'standard' bootstrap / installation tools like SystemImager, fai, JumpStart or Cobbler to provision a bare metal guest then hand it off to a configuration management system like Puppet, Chef, Cfengine to handle the rest.

  22. Canllaith said:

    I tend to use kickstart, and then have all of the post-install customisation done with Puppet or Cfengine. I haven't looked into Cobbler - but it looks pretty interesting. I might have to see if I can integrate it into the way I do things.

  23. Vallard said:

    We use xCAT and it handles ESXi 4.1 kickstart files (templating) very nicely. It's pretty full featured so takes over your DHCP, TFTP, DNS environment. The thing I love about it is that it also handles remote hardware control so you can install or reinstall nodes very quickly. xCAT is open source though it was started at IBM.

    I'd love to see a blog or article on using chef or puppet for configuring it after its installed with kickstart.

  24. Ian said:

    I use cobbler for PXE booting VMs extensively. We have a build vlan where our VM's PXE boot,and a base OS is installed. After the PXE process is complete, kickstart makes a web call to a little webservice i wrote, that changes the VLAN assignment of the newly created VM, to its "production" vlan.

    IMHO image based solutions like SystemImager or VM templates do not lend themselves to software updates very well. If you constantly are having to run updates on your base image that is an extra step in making sure you have the latest patch set.

    Kickstart allows you to perform a yum update as the last step of the pxe process, ensuring that you are getting the latest patch set for that time. Other machines can then be updated using chef/puppet, with a call to yum update.

    In short, image systems add another element of upkeep to your infrastructure.

  25. Thoughts on the importance of automated installs | Standalone Sysadmin said:

    [...] installations, to which I agreed. This means that I’m playing with kickstart and Cobbler again…but while I’m walking through this, I do have to wonder…why put much effort into [...]

Leave a Reply

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

*