Category Archives: System Administration

Posts generally related to system administration. This will be the primary RSS feed, and will include the vast majority of the posts, excluding those in Administrivia and some of those from Tech Field Day.

Debian Jessie and Puppet

Please correct me if this blog entry is wrong, because I really, truthfully hope it is.

It seems that Debian Jessie is not going to be receiving backports to Puppet 3 (client, anyway).  The way forward is through Puppet 4 (which most of you have probably known about for a while). Like me, you probably hoped to not have to go there so soon. Well, maybe not, but I had hoped to not go there so soon, anyway.

So the first thing I started to do was build out a Puppet 4 testing Vagrant box, which was surprisingly hard. Basically, puppet 3 supported the “–manifestdir” argument, but puppet 4 doesn’t, so if you install the puppet-agent package (which is how you install puppet 4 rather than puppet 3), it dies with “Could not parse application options: invalid option: –manifestdir”. Not cool.

The solution is to redesign your Vagrant puppet directory. Instead of “puppet/init.pp” and “puppet/modules/”, you need to make “puppet/environments/site/environmentname/manifests/site.pp” and “puppet/environment/site/environmentname/modules/”, respectively. The puppet config for my box looks like this:

config.vm.provision :puppet do |puppet|
puppet.hiera_config_path = "hiera.yaml"
puppet.environment_path = "puppet/environments"
puppet.environment = "vagrant"

The directory structure looks like this:

├── hiera
│   ├── common.yaml
│   └── nodes
├── hiera.yaml
├── puppet
│   ├── environments
│   │   └── vagrant
│   │       ├── manifests
│   │       │   └── site.pp
│   │       ├── modules
│   │       │   ├── collectd
│   │       │   ├── concat
│   │       │   ├── redis
│   │       │   ├── stdlib
│   │       │   └── sysctl
│   │       └── Puppetfile
│   └── Puppetfile.lock
└── Vagrantfile

Now, you should know… a lot of the puppet code will work without changing, but there were some not-small changes, including a few real head scratchers. For instance:

Empty Strings in Boolean Context are true

In previous versions of Puppet, an empty string was evaluated as a false boolean value. You would see this in variable and parameter default values where conditional checks would be used to determine if someone passed in a value or left it blank.

class empty_string_defaults (
  $parameter_to_check = ''
) {
  if $parameter_to_check {
    $parameter_to_check_real = $parameter_to_check
  } else {
    $parameter_to_check_real = 'default value'

Puppet’s old behavior of evaluating the empty string as false would allow you to set the default based on a simple if-statement. In Puppet 4.x, this behavior is flipped and $parameter_to_check_real will be set to an empty string.

You can check your existing codebase for this behavior with a puppet-lint plugin.

See the language page on boolean values for more info.

I… um… I don’t even really know what to say to that. I mean, they completely flipped the entire logic of the test around. That’s pretty unnecessary, in my book. I understand what they’re trying to say, that even an empty value isn’t False, but it’s hard to think of a more common shortcut when first writing code. I’m not going to argue that checking against an empty string is a good way to determine true or false, but wow, to change the behavior of the language entirely like that? That’s intense.

Another one I found that will probably kill code:

Regular Expressions Against Non-Strings

Matching a value that is not a string with a regular expression now raises an error. In 3.x, other data types were converted to string form before matching (often with surprising and undefined results).

$securitylevel = 2

case $securitylevel {
  /[1-3]/: { notify { 'security low': } }
  /[4-7]/: { notify { 'security medium': } }
  default: { notify { 'security high': } }

Prior to Puppet 4.0, the first regex would match, and the notify { ‘security low’: } resource would be put into the catalog.

Now, in Puppet 4.0, neither of the regexes would match because the value of $securitylevel is an integer, not a string, and so the default condition would match, resulting in the inclusion of notify { 'security high': } in the catalog.

So, I can kind of see that. A 2 is clearly a number. But on the other hand, the puppet code will happily cast the string ’30’ as a number because that’s valid. But the number 30 can easily be turned into a string, too: “30”. See?

I actually tested to see if you explicitly declared the “2” as a string, would the regex pick it up. Nope, not at all:

$securityLevel = “2”

case $securitylevel {
/[1-3]/: { notify { ‘security low’: } }
/[4-5]/: { notify { ‘security medium’: } }
default: { notify { ‘security high’: } }

vagrant@test-monitoring-client:/tmp/vagrant-puppet/environments/vagrant/manifests$ sudo puppet apply –modulepath=../modules/ ./site.pp
Warning: Facter: timeout option is not supported for custom facts and will be ignored.
Notice: Compiled catalog for test-monitoring-client.spacex.corp in environment production in 4.16 seconds
Notice: security high

Because of course not.

Anyway, I’m currently dealing with this, so I figured I’d write my first blog entry in a while so you could share in the fun. Good luck! And if you have a good technique for testing existing code against new puppet builds, let me know! I’m considering my CI options, but I’ve got a lot of new tests to write, it seems like.

Great Open Positions at Northeastern CCIS

I’ve landed in Los Angeles, and I’m getting settled in temporary housing until I find my own place, but it’s been a really busy couple of weeks, and I just realized that I didn’t get a chance to post about the open positions that my (now former) team has.

First, more obviously, there’s my old position, that of the Networking & Virtualization Administrator. The position is officially posted on Northeastern’s Careers page, but I can tell you that you’d be responsible for a medium-sized relatively flat network infrastructure. There are a few dozen VLANs, all statically routed from the core switches, and around a thousand lit switchports. The hardware is mostly Cisco Catalyst, with the core being Cisco Nexus 5548s, although there are some virtual PFsense boxes running around too.  You would be working with the (pretty friendly and competent) central ITS network admin to coordinate staff and faculty moves around the infrastructure, and with the university’s security officer (who is also surprisingly friendly, given his line of work) whenever something weird pops up.

The role is also responsible for the VMware cluster, which currently consists of around 15 ESXi nodes and two vCenter instances (one for “production” use which has the vSphere Essentials Plus license) and the educational cluster, built out using VMware Academic licenses for classroom and academic use. They’re backed by NetApp and Nimble storage, and it’s this part of the job responsibilities that gives you a little more creativity to solve problems, since professors usually want interesting things. I’ve built some useful stuff in PowerShell, but there’s no reason you have to use that long-term, if you want to solve the problems yourself.

Anyway, I really enjoyed my time in this position, and to be honest, I really miss the other staff members and students there.

In addition, the CCIS staff is growing. We got a new dean a little over a year ago, and one of the things she wants to do is to offer management of researchers’ clusters in a more active manner, so we are looking for another Linux sysadmin (pretty much all of the researchers do work on Linux).

This position will involve a lot working with our current Linux admin to bring over the technology he has built to deal with our “managed” machines to help with our “unmanaged” or “soon to be managed” researcher-owned machines. Basically, there’s nothing like this right now, so you would be inventing the role as you go. Exciting! Challenging! Rewarding!

Anyway, please, if you’re looking for a position in Boston somewhere, take a look at Northeastern. It’s easy to get to, there’s free tuition for you, your spouse, and your children, and I feel like the staff that I worked with there are my family, and I miss them :-)

If you have any questions, please drop me an email and I’ll be happy to help. Thanks!

Ad Astra Per Aspera – Leaving Boston

northeasterneduI really like working at Northeastern University, which is why I’m sad that I’m going to be leaving. On the other hand, life occasionally presents an opportunity to you that can’t ignore. This is one of those occasions.

A few months ago, I was sitting in a small room full of sysadmins planning LISA’15 when I mentioned, almost out of nowhere, that there was one company in the world that I would kill to work at. As luck would have it, my friend sitting next to me said, “Really? Because I know a guy. Want me to email him for you?” and I said, “Um, yes, please. ” Thus a story began that included numerous phone screenings, flying out to Los Angeles, and an all-day array of in-person interviews, the net result being that I am leaving Boston, moving to LA, and going to work…for Space Exploration Technologies Corporation, otherwise known as SpaceX. Yes, THAT SpaceX.



At SpaceX, I’m going to be a Linux System Administrator, and from the sounds of it, I’ll be splitting my time between “normal” infrastructure stuff and helping to define a DevOps role with the Flight Software team who write the software that sends the rocket and Dragon capsule up to the Space Station. It’s…pretty difficult to overstate how excited I am.


I imagine that it will take a while to figure out what I’m allowed to write about, but the whole team was very enthusiastic about my visibility in the SysAdmin space, and they seemed to enjoy my blog and the fact that I took part in the community, so I don’t think anything there will change. I’m just really happy to get the chance to do this, for a company with a mission like SpaceX. It’s an incredible opportunity, and I feel very fortunate.

So here we go, on a brand new adventure. I’m sad to be leaving my friends in Boston, but I’ll be back soon – I mean heck, LISA’16 is in Boston, so it’ll be like a homecoming, right? Until then, the sky is the limit! Keep reading!