September 2, 2010
For the past few days, our NYC office has had incredibly irritating problems with the internet connection. We’ve got service through a local Metro-E provider, but they’re a CLEC, which means they don’t own the lines, they just lease them from the ILEC, who is in this case, Verizon.
The root of the issue is that the wiring at the building we’re in is crap. It’s a small 5 story building that used to be apartments and has been converted to offices, and the wiring is just not up for the job. We went through several pairs of copper pairs looking for one that was good enough to carry the metro-E signal, and it was all we could do. Before metro-E, we had DSL, where we capped out at just over 1Mb/s…and this is in Manhattan.
Unfortunately, the circuit is currently in the middle of dying, so it’s working sometimes and failing others. I first opened this ticket on Monday, and have exchanged emails with our provider a dozen times or so. They’ll see the issue, but symptoms are vague as to whether it’s their equipment, our equipment, or the line running between our equipment, or (what I’m fairly sure the problem is), the lines entering the building from Verizon.
It wasn’t until last night when they finally saw enough errors on the bridge to have Verizon to commit to a service call tomorrow evening to add a loop. Every other time, everything on the line was hunky-dory. This is why intermittent problems take so long to solve…because all the stake holders have to be monitoring at exactly the right time for anything to get done.
Meanwhile, I’ve been having to apologize to my users, and give them instructions on how to forward their desk phones to their cells.
Even though the problem isn’t actually with my provider, I would love to get a secondary network connection, because the lines here are just too unreliable. No cable companies will give us service, no fiber companies will touch the building…it’s pretty much just Verizon and their CLECs at this point.
I think we’ve only got 2 more years on the lease?
Posted in General 
7 Comments »
September 1, 2010
There’s an amusing thread on the LOPSA Discuss list going on right now. It’s called “What Animal is a System Administrator“.
I was leaning toward the beaver until I saw the post by Paul Graydon, who recommends the Pooka, aka the Púca:
The púca has the power of human speech, and has been known to give
advice and lead people away from harm. Though the púca enjoys
confusing and often terrifying humans, it is considered to be
benevolent.
It’s like I’m looking in a mirror.
Posted in General 2 Comments »
September 1, 2010
The other day, I caught a message that KSplice was available for Fedora. I thought I’d be a wiseguy and I replied “Yeah, great. Call me in 20 years when it’s available for for RHEL”. Well, as several people pointed out, it turns out the joke is on me.
As you can see, it’s actually available for many Linux-based OSes at various prices. I suppose my confusion stemmed from the fact that I misunderstood what ksplice was.
My impression from a long time ago, when it first came out on Ubuntu, was that it was essentially a kernel patch that dynamically loaded patches and provided the ability to rebootstrap a kernel that was already loaded. As it turns out, it’s a commercial product that offers the ability to not have to reboot your machine to update the kernel. Let me be frank: I’m all about that.
The part that I kind of object to is in the press release, of all things. It’s the opening line of the company profile:
Ksplice is an enterprise software company making reboots a thing of the past.
Please, lets be honest. Reboots are inevitable. Using this product as a stop-gap for untimely reboots may be handy (at the low low price of $50 per year per server), but it can’t (and shouldn’t!) replace regular reboots.
The reasons for scheduled rebooting of machines are numerous. The primary one is that regular reboots assure that the machine is configured to boot correctly. If you’ve got a machine that’s got over 100 days of uptime, how do you know it will start correctly? You last booted it last quarter…what has happened to that machine since then? Changes in installed services, mountpoints, etc…it’s hard to tell if it’s going to be in a known-good state when it comes back up after a power failure.
Another reason to reboot occasionally is to clean up the running state of the machine. What’s that you say? Your machine is running fine? Well, sure, it may be, but how much cruft is left hanging that isn’t obvious? Have you ever used kill -9? Do you know for sure that there aren’t any memory leaks in your running services? Any processes hang while reading I/O and is now stuck in uninterruptible sleep?
Yes, there are lots of things that happen to servers over the course of doing their jobs. A reboot fixes many of them. The only argument against it is uptime.
I’ve written about uptime before, and I still feel the same way. Modern system administration has advanced beyond a single server providing a service. Uptime needs to be measured from the outside in, and according to the availability of the service, not the individual servers comprising that pool.
Feel free to disagree. Let me know if you’ve got an uptime of a year plus and you’re proud of it, or if you would be ashamed to be in that position.
Edit
This entry is causing quite a stir on Reddit. Cxunix from twitter also weighed in on his blog, servermanaged.it (link is in Italian, English translation here).
Posted in General 23 Comments »
August 30, 2010
This is apparently the “time to schedule your conference trips” part of the year, because there is news on the SysAdmin conference front.
First, and most pressing, the LISA10 conference schedule has been released! I’ve got to say, I’m digging the theme of the website, too. More important, though, is the content. Interestingly, all sessions and tutorials are available in half-day increments this year. This means that you can attend the first half of one session then migrate to another session after lunch. I’ve got mixed feelings about this, but I’m interested in how it will pan out. More flexibility is nice, though, and sometimes the first half of a session is really review (though there are a lot of arguments against that, too).
As always, there are discounts available for certain groups, and you do get a lower admission price if you’re a member of LOPSA, USENIX, or SAGE.
Check out the registration page for the fees. There’s an early-bird special going on until October 18th, so make sure you register soon. The return on investment for this conference is amazing.
I’m going to be there as a conference blogger, along with Matthew Sacks, Ben Cotton, and Marius Ducea. We’ll be publishing entries on the USENIX blog (which I’ll be linking to from here as well, of course).
Come to LISA and have a great time. And if you do decide to come, find me and say hello. I always love meeting readers.
Shifting gears a little bit, I’m sure you remember the PICC conference that LOPSA-NJ hosted. Well, we had a blast, and last year’s conference chair, William Bilancio, did an amazing job. It’s a bit much to do that twice in a row, though, so he was looking for someone to take the responsibility for this year’s conference, and after running it through my head a while, I decided that I’d take the job if he thought I’d do alright. Here’s his email announcing it:
It is with a great sigh of relief that Matt Simmons has decided to be
the Program Chair for PICC ‘11.
Last year Matt was the head of the marketing team and did a great job
at getting the word out about the conference and was a key person in
making last years conference a success.
Tom and I feel that he will do a great job as the Program Chair and
will make PICC ‘11 a great conference.
In other news I will be getting in contact with the hotel and get the
date locked in, in the next few weeks and then we can start really
working on the conference.
Please start thinking about sponsor ideas as well as any new people
you think will be able to help make PICC ‘11 another great conference.
Again thank you Matt for taking PICC ‘11 Program Chair job and good luck.
William
I want to thank William and everyone who was involved with last year’s conference. Everyone I’ve talked to had a great time and has been looking forward to this coming year. I’m going to work hard to try to improve on William’s example, and really grow the community of system administrators in New Jersey and the rest of the northeast. I’m going to need help, though, so if you helped out last year, I’ll be calling on you now. If you weren’t involved last year, now is a great time. Drop me an email or comment on this story to let me know that you’re interested in volunteering. We can definitely use the help.
In addition, I was talking to Lee Damon, who let me know about a SysAdmin conference called “Cascadia IT Conference” (aka “CasITConf”), and it’s happening in the Pacific Northwest. It’s being put on by SASAG, the Seattle-Area System Administrators’ Guild.
So there you go. Three sysadmin conferences in one post. It’s going to be a busy year for everyone, so get involved and lend a hand to someone in your area!
Posted in General 7 Comments »
August 27, 2010
My datacenter migration (or renovation, as I’m referring to it) includes a fair amount of added virtualization. We’ll be maxing out the memory and processor power of three machines at each site, and those will act as a VMware HA cluster (we’re buying the vSphere Essentials Plus license kit for each site).
Of course, I’ve got to have some VMs to run. I could reinstall all of my machines using cobbler (which would invoke the gods of trial and error, not to mention incur Murphy’s Wrath), or I could convert the machines that already exist from physical to virtual (p2v). That second option sounds much less error prone.
That being said, converting a physical machine to a VM isn’t exactly a fast process. Hoping to get it done the weekend of the move would be foolish, so I need to get it done beforehand. That’s why I’m driving to Philadelphia today.
Last week, I threw a couple of terabyte SATA drives into a spare PowerEdge 1950 server, upped the RAM a bit, and installed a freshly minted copy of vCenter Hypervisor 4.1 (formerly known as ESXi). I’m trucking this machine down to our secondary data site today so that I can begin the p2v conversion process. I’ve got enough disk space that I won’t run out (I’m only putting the root partitions in the VMs, since all the data is stored on the SAN), and I don’t need to actually run the machines, so RAM won’t be a problem. This will just be a holding tank until I get the VM hosts setup during the conversion weekend.
The actual conversion will be done using VMware Converter, a free tool by VMware that I’ve been really impressed with. It does want an ESXi…err..vCenter Hypervisor server to connect to, but that’s free too.
Once this is down there, I’ve got some decisions to make. Namely, I need to decide how long to wait until I do the conversion. Not a lot of data changes on the root partition. It’s going to be limited to logs, really (since I haven’t gotten a centralized syslog server running yet). The exception to this rule is the domain controller at that site. That needs to be the absolutely last machine I convert, and once I do it, I’ve got to turn off the source, because if the image becomes too far out of sync, well…that’s sort of like crossing the streams.
So, has anyone else pre-converted VMs like this in preparation for a move? Any advice or caveats to watch for?
Edit
Fixed the mistaken Ghostbusters quote. Did I seriously say “crossing the beams”? I am disappoint.
Posted in General 6 Comments »
August 26, 2010
Alright, several people have asked me why I haven’t weighed in on the current “devops” movement. Mostly because no two people can absolutely agree on what DevOps is. I’m outside of that particular community, although I read a lot of the blogs of the key members, so maybe I’m in a good position to comment on my perspective.
First, lets define DevOps. If you strip away all of the touchy-feely stuff that gets associated with the name, devops is, at its core, DevOps is an increased interaction and interdependency between developers and operations staff, whether that operations staff is specifically system administrators or whatever.
This means that the people who develop code no longer have willful ignorance of operational environments, and the people who operate the environments can’t do so in a vacuum of knowledge about the software itself. This increased communication and reliance IS DevOps. That’s it. Nothing more. It’s a methodology. It’s not a panacea and it’s not for everyone. How can you tell if it’s for you?
Let’s answer some questions…
- Does your organization have programmers?
Developers are necessary for the DevOps relationship…otherwise you’ve just got Ops
- Do you provide Software as a Service?
DevOps grew up in the web world, around places like Flickr, who provide applications over the web. Other people may just think of them like websites, but in actuality, they’re applications with incredibly large code bases. Since a solid application depends on well-developed code running in a known stable environment, it’s natural that this kind of biosphere would produce methods like DevOps
- Do you release software updates frequently?
If you’re in an environment where something is broken and gets fixed immediately, then you can say yes here, but it’s not just bug fixes. Features get rolled out, pulled in, and switched around. Agility of this nature isn’t possible without everyone working from the same playbook. It’s also not possible with an environment that can’t change rapidly to match the code.
For the 90% of companies out there without that particular environment, then you probably aren’t using DevOps, and that’s fine, because there’s almost nothing it can do for you. Especially if you don’t have programmers. Because hey, no dev, right?
You’ll notice that nowhere in the preceding text did I mention the tools that DevOps uses. That’s because the tools are completely separate. Using “puppet” doesn’t mean you subscribe to the DevOps methods (or even the mentality), and although DevOps may not be necessary for your environment, you might find puppet extremely useful. Let me say that again, Using the same tools as DevOps shops use does not tie you to the DevOps methodology.
As alluded to in the last answer up above, the shops that run DevOps need environments that can change quickly and absolutely. They needed tools that could do it, because you can’t manually change hundreds of application servers. Because of their need to change that many machines, and have it happen nearly instantaneously, tools to automate this kind of change were developed and implemented.
Other technologies that get lumped into DevOps, cloud computing and virtualization, are also natural off-shoots of the type of environment where you have hundreds of application servers. Of course that kind of environment is going to be heavily into virtualization (if they’ve got an existing large infrastructure) or cloud computing (if they don’t).
Again, DevOps doesn’t “own” these technologies. They just use them (and advance them by writing tools to improve them, in many cases).
So there, that’s my take. For the people who can use it, DevOps is developing into an exciting methodology to ensure increased availability and stability of IT resources.
It’s not for everyone, but you owe it to yourself to take a look at the tools that too many people have been misbranded “DevOps”. There’s a lot of functionality there, and it can decrease the amount of time you spend slogging through administrative tasks.
Edit
It looks like I’m not the only one who’s been thinking about this, too. Benjamin Smith wrote his take as well, and it seems like we agree quite a bit.
Posted in General 3 Comments »
Email me
content rss