Puppet, PICC, and more updates
May 6, 2010
For the past couple of weeks, I’ve been diving into some fascinating rabbit holes in regard to puppet, which I’m still haven’t implemented in production (but I’m getting closer!).
I don’t have time to do a full write up of what I’ve gotten done, but essentially, I need to be able to be able to roll out a “type” of machine (in this case, a machine that produces reports). Normally, I’d just do a kickstart install, and specify the packages to be installed (which would include puppet), then manage the services via puppet, but our report generating machine takes all sorts of strange and customized software that historically is just rsync’d into place. Well, puppet doesn’t like that sort of thing.
I ended up writing a script that mounted an NFS share, rsync’d the files across, and then cleaned up after itself. There were a couple of problems with this.
- I lost an entire day to the script not working
- This is totally the wrong way to do it
Basically, the script ran FINE by hand, but when puppet ran it, no dice. Me and nearly the entire population of #puppet on freenode worked for hours trying to figure out what it was. Eventually, one of the channel members suggested that I run strace on the entire process. One little tiny error line led to me figuring out the problem. “permission denied” when running fstat() on the NFS mount point…and it happened regardless of whether the NFS share was mounted. After doing some deep thinking, I ran getenforce. Yep, SELinux was turned on. /sigh
As hard as I worked to get my script to function, at the end of the day, I had to look over it and admit that I wasn’t doing it the right way. Instead of rsyncing all of these custom installs into place like we always had, the best way is to install them as a package. Deeper and deeper into the rabbit hole.
In order to install something via a package, you’ve (obviously) got to have the package. Which meant that I had to make the package. Bother.
There are dozens of instructions for creating RPMs throughout the internet. I used a combination of them. I really liked Packaging Software with RPM from IBM, but Troy Dawson from Fermi wrote an excellent introduction using Pinky and The Brain. Awesome stuff.
I’m smack in the middle of getting my RPMs ready to be distributed, but to do that, you need a distributor. When it comes to RPMs, that means you need a YUM repository. Fortunately, that’s going to be the easy step. Apparently, all you really need is to run createrepo on a directory that’s available via http. Once that step is completed, you just add the repo to all of your machines. Fortunately, by the time that’s necessary, puppet will be up and running.
If you’re not familiar with puppet, there is some good information at the official doc site. I actually bought Pulling Strings with Puppet
As a coincidence, Jes at canllaith.org talked about Simple Steps with Puppet today, too.
Anyway, that is what I would be working on otherwise, but I’m getting things straightened out for PICC, which starts tomorrow. It’s going to be a lot of fun, and I’m looking forward to seeing several of you there.
Incidentally, with PICC< we're trying something novel with the conference blog. Essentially, anyone attending the conference will be able to publish entries on the conference blog (I'll post the link as soon as I have it). Watch for tweets with the #PICC10 hashtag!
Posted in General
Email me
content rss
May 6th, 2010 at 9:29 am
[...] This post was mentioned on Twitter by Matt Simmons, Planet SysAd. Planet SysAd said: Standalone Sysadmin: Puppet, PICC, and more updates http://bit.ly/d4Wrpp [...]
May 6th, 2010 at 9:41 am
I actually have just started using puppet a few weeks ago. Right now, when I build a new machine I simply install puppet first. After getting the certificate right with the puppetmaster the new machine installs some basic apps. (debian, so they apt-get it)
Puppetmaster tells the puppet to install build-essential, fail2ban, sshd, vim, logcheck, rkhunter, ntp, sudo, rsync. I have each of these built as separate modules for configuration clarity. As part of each module, I also pass down the corresponding already set config. Puppet puts the config in the right place and the new machine is up and running.
May 6th, 2010 at 9:48 am
Thats great Matt. Hey while you are setting up that repo if you haven’t done so already you should setup local mirrors. Checkout mrepo, it can mirror for both Centos and RHN.
May 6th, 2010 at 11:45 am
Lee: That sounds like a cool setup. The more I play with puppet, the more things I see that will be excellent for it.
Nick: wow, I hadn’t thought of that, but it would definitely save bandwidth over the long haul. Good call!
May 6th, 2010 at 1:15 pm
The most awesome book on boostrapping a new linux/unix infrastructure:
http://www.amazon.com/Automating-System-Administration-Second-Experts/dp/1430210591
It explains how to jumpstart/kickstart/fai-start installations and manage them all with a cfengine policy host. Cfengine is awesome, the book is great.
May 6th, 2010 at 1:48 pm
It’s a good idea to design your Puppet infrastructure around either of these tools.
http://theforeman.org/wiki/foreman/Screenshots
http://github.com/reductivelabs/puppet-dashboard
They enforce some good design principles, especially with regards to modules+classes, grouping your servers and assigning custom variables for hosts. Foreman tries to do many things besides puppet, but is definitely more usable right now.
They are also invaluable for reporting(via email and web), when you have hundreds of puppet nodes.
May 6th, 2010 at 4:36 pm
I sure did! Although to be honest that post has been sitting in a txt file in dropbox for about 6 months waiting for me to have the time to polish it up a bit and post it.
I’m about to start a new job that is going to involve going back to some Cfengine I believe – take pity on me! Lets hope I’m allowed to puppetise it.
May 7th, 2010 at 2:32 am
@Jes:
I am in now way linked to cfengine.com; I just use their open source product extensively without paying for support.
it seems trendy to bash cfengine now. To be honest, I do not see why. It is a robust tool that does what it has do fast without getting in the way. It has a clear syntax, a friendly and helpful community and if you need to burn some cash, you can get paid support if your bosses require that sort of thing.
At one point there was a really funny piece at the reductivelabs site plenty of FUD about cfengine. It is offline now, but still readable through archive.org: http://web.archive.org/web/20080526101643/http://reductivelabs.com/trac/puppet/wiki/CfengineVsPuppet
All the points named in that page but one (code reuse) were at the point the page was written simple lies (the code reuse point has been addressed in cfengine 3). Yet people do not check them and assume that they are true. i guess Luke Kanies got bad feedback about that when he went to shops that were already using cfengine and took the site offline. If you have a good product, you do not need to lie about a competitor, that reflects bad on yourself.
May 7th, 2010 at 2:58 am
natxo:
Thanks for the information. I think it’s important that the truth always come to light.
Honestly, though, I have no thing against either of the pieces of software (or chef, for that matter). I just had to select one, and I went with puppet because I thought the barrier of entry was smallest there. I could have flipped a different coin and gone with cfengine. Both have large userbases that are apparently able to manage perfectly fine.
I don’t think Jes was bashing so much as teasing. I’m sure that, if given time, a holy war will emerge from the stress between whichever two configuration management systems end up going the longest. I suppose there’s no reason that all three can’t continue indefinitely, but it’s so much easier to hate your enemies intensely when there’s only one of them to concentrate on. (We have always been at war with Eurasia).
As I said, it’s important that the truth comes out. Whether Luke was genuinely mistaken or intentionally misleading really only reflects on him as a person, not on the software. The same would go for cfengine if Mark were to do things like that.
An idea isn’t responsible for the people who believe in it.
But thanks for the link and the comments! Please keep them coming!
May 7th, 2010 at 6:29 am
Instead of making packages (if you really don’t want to), you could have things checked into some version control somewhere and update checkouts from that using puppet.
I’ve used this for dev/test/production versions of websites and some customized software, as if you’re going to modify things then you want it in version control anyway, and making packages from it every time seemed a pointless extra step. (Branches for dev/test/production are also your friend in this case)
I should write up this, as I’ve got it working fine for SVN and Git repos, and it’s nice to just run Puppet and get the latest code on a branch to the production server without any intermediate steps.
May 7th, 2010 at 11:00 am
Similar experiences to this are why one of the first things our cfengine setup does is disable selinux! We can selectively enable it on a per-server basis if we choose, but off by default means we get to keep our sanity.
May 7th, 2010 at 12:06 pm
@Pete: and that’s why I took Rik Farrow’s SELinux training last year at LISA. The course has allowed me to have it left enabled on four servers so far…yeah, small shop, and that’s still a small part of the machines I administer, but it’s a lot more than I had before.
I sincerely hope he’ll a) be doing the course again this year and b) he’ll write a book about SELinux. I can’t recommend it enough.
May 8th, 2010 at 5:37 am
@natxo asenjo, I think you’re mistaking me for someone who represents puppet. I’m a systems consultant who’s used both products in the course of my work and I don’t have any ties to either project. No validity to any accusations of spreading FUD here – no-one would listen to me if I tried =)
Cfengine was my first choice a few years ago – it’s an excellent product – but my programming skills being weaker than some I find puppet’s syntax to be much easier and clearer to deal with. This is a completely subjective opinion – without having my exact programming background (or lack of one ;) a person may not agree that it’s easier.
My apologies for not being clear as to why I prefer puppet – I can see how someone could interpret my comment as trendy puppet-rocks-cfengine-sucks bashing and that certainly wasn’t what I intended.
August 30th, 2011 at 9:14 am
both products have pro’s and con’s, but I think ‘il stick with Cfengine, due to the complete lack of file edditing posibilities in puppet and the overall maturity of the cfengine software which is much better.
Other thing is performance, since I mostly work in big shops perfomance is an issue, my last issue with puppet is that I need to maintain a complete ruby stack on all machines which is to my opinion undesirable.
Cheers,
Patrick