Just what we need...another package manager

Date March 18, 2014

The rust programming language announced the release of Cargo, their new package manager, which will, "support the common lifecycle for packages", because we don't have enough of that already.

Before I start, let me just say - I'm not picking on rust or cargo. This isn't about them. This is about the ecosystem.

We have too many package managers. Far, far, far too many package managers.

Each Unix-like OS has their own package management solution. And each programming language seems to have ITS own, too. Whether it's CPAN, or pip, or pear, or easy_install, or gems, npm, or now cargo, there's a way to install addons for software specific to a language, in that language itself, but more importantly, outside of the OS package management.

And then, there are other layers to contend with. DSLs have their own internal way of installing modules. Puppet has the forge, chef has knife and the community cookbooks, and CFengine has the..well, whatever it has, I'm sure. Then there is meta-software to manage those, too. Things like librarian-puppet, which is, itself, a customization of a piece of software called librarian, which is a framework to manage bundlers, which is something that manages ruby dependencies (pulling down the software you need to run the things that you have). There is also librarian-chef. This is separate from Berkshelf, of course, which does pretty much the same thing.

How in the ever-loving sight of his noodley appendages are any of us supposed to manage this heaping layer of crap?

Yes, I know, some of the aforementioned software solutions are meant for managing a single user's module directory maybe, or a specific dev environment, or for testing. But which, and why, and is it explained anywhere that the software should or shouldn't be used in a certain environment or way, or are you just given the sticks, the twine, and the plastic and told to go fly a kite?

The irony of all of this is that we're using this jumbled up pile of random software which largely overlaps in purpose and function, to try to build more solid, easily reproducible infrastructures. But rebuilding the environment that created it? Good luck!

Maybe it's not all that bad. Maybe. I mean, after all, I have the artifacts that were created, right? I have the puppet code that I used to produce my infrastructure. Well, sort of. I mean, to be really honest, I don't write all that much puppet code. I use off the shelf puppet code, and I end up writing YAML for hiera to interpret and feed into parameterized classes. So my artifacts aren't puppet code so much as really ugly stuff that isn't markup, according to the definition of the language construct.

So how do I rebuild the compost heap infrastructure that I used to build my environment? Suppose I used entirely off-the-shelf puppet code. Nothing custom, just modules I found. And I erased my repo which contains my puppet modules. How would I rebuild it and get the same thing that I had before?

Do I stand a chance in hades of actually being able to reproduce what I had before? Suppose I even had a list of all of the puppet modules. Could I source the same modules? The same versions of the same modules? No. And I'm willing to bet, neither could you.

If you're like me, you've probably cobbled together some kind of unholy array of modules from the forge, modules from github, and modules of ill repute that you found in a back alley somewhere, and that's all stuff that you probably think of as "off the shelf". It's completely irreplaceable, if only because there's no good way to figure out where what came from, because I (and most likely you) used any one of a dozen methods of getting the module into the system.

I don't think there's any hope for fixing the 'bespoke build infrastructure', at least in this generation. We have to deal with the fact that the systems we use to build our highly available, fully reproducible, solid infrastructures are, themselves, anything but.

However, if I could plead to the tool makers, to the developers who are, have, and will build the things we use to build things, let me ask you this. Please, swallow your hubris. Please, accept that a tool someone else wrote that you see as imperfect may actually solve the problem, and know that the world may not need another solution that does the same thing. Instead of building something new that overlaps, work with the other person to extend what they've done, don't build anew.

Unless we learn this lesson, we're going to continue to have infrastructures that look like the Walled City of Kowloon, rather than something better. Please.

  • Pingback: Packaging for Systems | Bob Martens()

  • skyborne

    It's an inductive problem. You will always have a starting point that can fail. If you're worried about losing your puppet configuration and you write anything that can re-create it, you are now at risk of losing that. Maybe I use packer to provision Fedora Cloud instances; what happens if the Fedora Cloud spin goes away or I lose the disk image?

    The fact that each language has its own package manager (written in that language so as to bear some semblance of being "dependency free" on its own) is an entirely separate problem, but I don't foresee anyone ever caring enough to distribute RHEL 5+ compatible static binaries of a package manager that understands every new hotness (bundler, carton, leiningen, bdist/egg/wheel, composer, lua rocks, etc.)

  • KaeseEs

    You make some good points. Another downside of cargo is that it's made by rust programmers and thus it's likely to be competently constructed and unobtrusive - we won't get any comedy gold from it like we get from npm on what seems like a weekly basis.

  • Foo

    Kowloon Walled City, not "Walled City of Kowloon". I know. I used to work there. It's gone now.

  • Chris

    And then the question becomes "oh haggard sysadmin, which package manager to rule them all? Which one shall we all use? I hope it has bindings for the language / environment I'm working in!" The closest thing we have right now is fpm, which is a metapackage manager.

    Putting on my developer hat here, I would say the reason why we have so many package managers is because programmers want to use the language they're best at to programmatically interact with said package manager. Rust programmers like Rust. Perl programmers like Perl, and so on. This creates the environment of wheel reinvention that you're seeing. It's not hubris. It's convenience and speed of development.

    It's for that reason that I believe there never will be a centralized package manager, unless it can be written in a way that it provides APIs in every language people want to use it with and supports all the particularities and idiosyncrasies of those environments as well. This is quite a task, and the skills involved (fluency in several programming languages, knowledge of several operating systems' package managers and their quirks) typically require the sort of developer and admin cooperation which is rare in software.

    It's harder to say at the operating system level, though. It could be as simple as licensing--back when all the major OS package managers were developed, GPL vs. not-GPL-but-still-open-source was a big deal. These days, with more software moving away from the GPL, this is becoming less of an issue (thankfully!). For the reasons stated above, though, I don't particularly think that we're going to see something unified any time soon. Add to all of this the inertia of the current systems. Even within a programming language community it's very difficult to introduce new tools that work with the current system. Devising an entirely new system is generally impossible because. Inertia doesn't build linearly, or even quadratically. It's logarithmic, at least. If you can find a way to defy "IT physics", maybe you'll have a chance of introducing a unified package manager. Maybe you'll even win a Nobel prize.

  • carlivar

    Deploy software as containers. Let developers use whatever package management methodology they want in the container as long as certain standards are met regarding logging, performance metrics, monitoring, etc.

  • cbarbour

    A big part of the reason we have so many package managers is that package management is a hugely important aspect of infrastructure management.

    I do puppet professionally, and I've never felt that we have too many package management solutions; on the contrary, I usually finding myself wanting additional options to handle special cases.

    "We need to bring in Graphite to handle package deployment on the Windows Infrastructure." "It would be nice if there was a way to package vendor installers without violating the vendor's support requirements."

    Puppet especially, but the other vendors as well do a good job at abstracting the complexity of using different package management solutions. Generally, the main complexity is handling platforms that don't have a repository system, and handling different naming conventions and installation pathing used between vendors; a problem that a unified packaging solution wouldn't really resolve.

    'puppet resource package ntpd ensure=installed' works on pretty much any platform.

  • http://be.groovie.org/ Ben Bangert

    It's useful to remember that this is not like npm/easy_install/gems because Rust is compiled. Your life as a sysadmin is to take the binary and put it on the machine however works best. The package management for Rust is merely for making the build environment which the developer/build-system does.

    Maybe that involves packaging it up in a .deb/rpm/whatever, but it can be treated as the end-product like something out of any static language. Confusing it with the mess ops typically has to deal with when trying to deploy dynamic languages isn't that fair.

  • http://shokk.wordpress.com Ernie Oporto

    Don't forget, Cargo could be a tool someone else wrote that you see as imperfect, and it may actually solve the problem.

  • José Dinuncio

    Hi, Matt

    I've thinking about it, and I share your opinion.

    But, what can we do to improve the situation? Since most distros probably won't change their package manager, at least one package manage per distro seems to be the most plausible acceptable solution.

    Maybe it could be possible to develop a package description language that could be used to generate .rpms and .debs and evangelize for most languages to use it to describe their own packages?

  • Luca Bruno

    Take a look at nix: http://nixos.org/nix/