Dear .Edu People: How are you handling external cloud fees?

Date December 2, 2013

Working for the College of Computer and Information Science is definitely a challenge in certain ways. The people that I deal with are almost universally computer-literate and very skilled and knowledgeable about what they're doing, and they're all doing different and interesting things.

While I do provide a virtualization environment for general purpose classes (under VMware), I'm seeing more and more interest in spinning up instances of student VMs in Amazon (not to mention the classes that are doing things like running Hadoop in the cloud so that they don't need to micromanage their MapReduce infrastructures). Right now is probably the most flexible time for computing I've ever seen. You can just about run whatever you want wherever you want, if you have the funding.

But that's kind of an interesting problem in and of itself. Suppose that we, the college, did have the funding, centrally, and not per-project, or per-class, or per-instructor. Or, even, suppose we have it per program. How do you equitably divide that up among the researchers, instructors, classes, and students so that it gets to the people who need it most?

This is the meta problem that I'm considering right now, and I know I'm not alone in being concerned that the amount of money that we're using and the amount of money that we've got budgeted might not line up in any meaningful way. If you have an environment where several people can spin up instances at a whim, how do you ensure costs don't go insane? Or, alternatively, how do you control the resources that someone can instantiate?

That's what I'm asking. Are you using an external cloud provider? How do you control your scaling? How do you make sure that you don't suffer a Pyrrhic victory in terms of traffic and usage? What are you doing to manage this kind of thing?

Please comment below!

  • http://serverfault.com/ Michael Hampton

    You bill for it, of course.

  • Scott Pack

    The answer to your first question is a resounding, "Of course you are whether you know it or not." Everything else will fall out of making sure the clients understand what they're getting into. Be sure to ask questions about what kind of data is going up. Is it regulated? Is it classified? Is it covered under any contractual agreement? Do they actually have the authority to put the data up there? Once they start thinking along those lines get them on the topic of availability. Are there any SLAs involved? Are there any availability concerns that might warrant proper DR considerations? When the time comes what are the provisions and methods to actually get their data back? That's when you should start asking about, and talking to the person, who will manage the systems. As much as possible they shouldn't be treated any differently than what's on-premise. Make sure someone is designated as a contact person and that, if you have any, it's configured and managed in accordance with your standards and policies. Some changes will have to be made of course, but understanding which and figuring out the right methods are the fun parts, eh?

  • Scott Pack

    I suppose I should have mentioned somewhere in there that you should probably work with the relevant business person to determine a pricing model. Either they pay for it outright and you're put in the purchasing chain as an approval gate, you all handle it and do a charge back, or absorb it as a base service and figure out how to get it to work in your budget preferably with a proportional increase to base funds. That's going to be a rather unfun conversation with your Budget Unit Manager or whatever.

  • http://www.standalone-sysadmin.com Matt Simmons

    Scott: Thanks, great comments.

    Assuming you work out the monetary responsibility dotted lines, what do you use to ensure that a department or $unit doesn't go over their allotment?

  • Nick Cammorato

    We have the projects handle it. One of the big problems historically has been PIs actually getting the relevant infrastructure units involved when writing out the proposals in order to assess what they actually need(and what to budget for) though. This has gotten a lot better in the past few years, but welp, welcome to academia!

    The dividing line for whether we pay for it or are a hard gate is what percentage of projects are using it. If it's over 60%, then we absorb it.

    A lot of this is down to department/organizational structure though, so I'd be curious to know more about how NEU handles that inside the departments(I would like to be Puppet CFO).

  • Scott Pack

    Matt, that's a tricksome one. I imagine it would have to depend quite a bit on the platform you go with, what tools they have available, and how good an accountability you have with the finances. Based on my experience working in education I don't envy you having to figure out how to enforce such things.