I am my own worst enemy sometimes
July 8, 2009
You’ve heard the phrase, “haste makes waste”? I don’t think I actually understood the concept until yesterday. I had a horrible day, and it was entirely my own fault. The worst part is that I made it miserable for other people, too.
We recently hired two new salespeople. Yesterday was their first day, and their laptops weren’t in yet, so I took a couple of spares we had and prepared them. Or at least, I thought I did.
One of the laptops had been a “floater”, a spare that anyone could use if they needed one, or for visitors from another office. That one was always kept up to date, so I knew it was fine.
Another had been left over from someone else leaving, so I wanted to reimage it. Of course, we don’t actually /have/ images of any of the machines yet, so I reinstalled it from scratch. And I was in a hurry, so I decided to do this on my desk, where I could continue to work on my other stuff. Just consider this foreshadowing.
I installed Office, plus all the software the salespeople would use, and then installed all of the patches and got the software up to date, and then I created an image of it using Clonezilla (it was the fastest solution for this, from what I saw). Then I added it to the domain, and made sure I could log in.
I wanted to be thorough, so I asked another domain user to log in. He was successful on the first time, so I thought “aha! It’s ready!” I alerted the guy who was going to deliver the laptops to their new users, and went back to the stuff I was doing before.
Fast forward to yesterday. The user of the laptop that I reinstalled couldn’t log on. I did some basic troubleshooting, and then I stepped back and thought to myself…this is only a sign. If something is screwed up badly enough that she can’t log in, then the machine is going to be enough screwed up that I should be on hand for the carnage.
I told them to give me an hour and a half and I would be in the office. It took me around 2 hours , just because the train pulled away as I parked my car.
It’s too depressing to list all of the things that went wrong yesterday. If you want a symptom of the bigger issue, the reason she couldn’t log in was because when I tested on my desk, I didn’t have a spare ethernet run. Rather than move the laptop to the server room and do it the right way, I connected it via wireless…and forgot to install the NIC drivers. That should give you some idea of what I put myself (and those poor new employees) through.
So yes, I’ve been working on developing a comprehensive “new user” checklist that I’m going to adhere to, if I have to print it out and thumb tack it to the walls of my desk. If you don’t have one yet, I suggest you consider it, and save yourself (and your new hires) from what I put mine through
Posted in General
Email me
content rss
July 8th, 2009 at 9:54 am
Been there and done that before! Also the “New User” check list is required for what I need to do.
Hopefully today is better than yesterday and you have learned from your mistakes.
July 8th, 2009 at 10:18 am
Second this! Build Sheets are good at deterring this…. but when having a bad day…
Let me know if you want our template (New User or Build Sheet), Matt.
July 8th, 2009 at 10:27 am
@Steve – thanks. I have, and I’ve got the list complete (for now)
@John – That sounds great. If you send them, do you want to share them with everyone else? I can host them on here.
July 8th, 2009 at 10:39 am
Sounds like a bad day alright, but there’s always a lesson. I’m glad you’ve learned yours.
I find checklists very helpful as well!
July 8th, 2009 at 1:07 pm
Yay, I started with build sheets too, and then as time progressed I turned these into custom images on a Terabyte drive, which soon turned into creating custom Deployement Images for WDS. Now I just PXE and click click and the image is pushed out over the net, super simple, saves me even having to go to the department to do a Fresh install
July 8th, 2009 at 1:36 pm
You need an unattended installation routine.
July 8th, 2009 at 1:40 pm
@noneck: I can’t wait until my auto-install infrastructure is in place
@Natxo: absolutely. I actually wrote about that not too long ago: http://www.standalone-sysadmin.com/blog/2009/06/windows-desktop-automated-installations/
July 8th, 2009 at 3:28 pm
Sorry to hear about your troubles….I think anyone that reads this blog has been there before including myself. I guess some comfort can come from that your not alone hopefully.
July 8th, 2009 at 3:42 pm
Thanks AJ. I never try to hide my mistakes on here, just because if someone else can learn from them, then something is gained.
http://despair.com/mis24×30prin.html
July 8th, 2009 at 7:12 pm
Yeah Matt certainly been there many a time myself – I think the checklist is the perfect solution, getting a “run book” or at least some sort of documentation makes things so much easier and more organized in the long run. Glad it wasn’t anything serious!
July 8th, 2009 at 7:42 pm
No need for a check list. Just find some time to create a standard image. It’s definitely worth it.
July 8th, 2009 at 8:40 pm
@Anthony – Thanks. I had to look up a “run book”, but yeah, that sounds pretty much like what I’m creating. Just a series of instructions to cover some non-scriptable procedures.
If you can’t script it, use a checklist.
@Chris – I agree, an image will help a lot. Lots of these problems were related to the actual accounts, though, plus the issue that these machines were essentially “one off” installs, since they’re pretty much the last non-standard machines we’ll be getting (hopefully, anyway). When we do get the standard machines in, they will be created using an imaging system. I can’t wait to see the amount of time that will save me.
July 9th, 2009 at 1:32 am
The idea of “automate what you can, make a checklist for what you can’t” really appeals to me, if for no other reason than to avoid these sorts of embarrassing mistakes in the future and to at least feel a little bit more organized.
Habits like this are easy to adopt if you’re a “standalone” sysadmin, but here’s my problem: I work in a small team where the boss is just a little bit too laissez-faire about the whole thing, and my coworkers may be a little bit less inclined to do something just because I say it might be a good idea.
We can get these sorts of ideas and improvements to take if we can get invested in it a little bit, like we did by purchasing a monitoring system and training instead of using Cacti. You sink a little money into the project and all of a sudden, there’s some real traction and excitement about it. Can you tell I’ve been burned spending one too many late nights configuring apps (wiki, ticket system, etc) that never REALLY got used by my team or anyone at all?
Matt, where did you originally hear the “script it or checklist” concept? In some particularly interesting talk? That might help…
Does anyone have a suggestion on how to get coworkers excited about getting organized? It truly takes a team effort to keep these sorts of things up to date and working.
July 9th, 2009 at 1:41 am
A standard image gets 80% of the work done in 20% of the time. Three cheers.
But in a lot of orgs, there’s more to it in a lot of places than blasting out a fresh windows install. Additionally, we need to set up the user’s phone, a handful of non-LDAPpable accounts, and that sort of thing. Phone provisioning can be difficult to automate because in current practice, it requires a service call, for example.
July 9th, 2009 at 3:17 am
Everyone fall in this, I guess. I falled on it before, As we have almost 55 users in site and from 3-5 users monthly are transfeer or updating.
Checklist was a must to have,
standard software for each users type (Commercial, HR, Decoraion ..etc)
Computer settings (Domain joining, drives installation ..etc) .. and so on.
Good subject to talk in but bad trigger for you Matt to bring it active.
July 9th, 2009 at 8:39 am
@Kaput
Please check out today’s entry
July 20th, 2009 at 4:35 pm
Arr, we’ve all been there and done that dude – even with a checklist you can be having the worst day and still miss something sometimes. It’s, sadly, just human nature. Even sysadmins aren’t infallible!
July 20th, 2009 at 4:39 pm
@the_angry_angel
Thanks for the support. I know it too, but it’s easy to forget about it when you’re the one that’s on the line
Come back and visit anytime.