Dear Lenovo: Love is over.

You know, for a company that named their product the “Thinkpad”, they didn’t really take it to heart.

A few months ago, we ordered some Lenovo Thinkpad T510s to act as stock for new employees that we were hiring. They just sort of sat in the boxes until we needed them, and when that time came around, we have a process we go through to prepare them with the standard image.

Ryan, my junior admin, was taking care of that, and like a good observant sysadmin, he noticed that one of them was running warmer than the rest. Investigating further, he noted that you could see the heatsink through the vents on the side of the machine, and it was obviously crooked, which was not the case in the others. Clearly, it had come off in some way.

Since I don’t relish the thought of disassembling laptops that are still under warranty, we called CDW. It had been over 30 days since we bought them, so there wasn’t a lot they could do, other than refer us to Lenovo. No problem, we got the enhanced on-site repair warranty with the depot shipping option. The call was made, the appointment was scheduled, and the repair was done. Success.

Except…once the repair was made, Ryan noticed some other disturbing tendencies…while transferring the image, he noticed that whenever he bumped the machine, Clonezilla‘s Linux terminal displayed a USB disconnect kernel message. Then there was the slight fact that the machine didn’t sit exactly flat…it sort of wobbled, like a table with a short leg.

Obviously, something was wrong with the machine. It had apparently not been assembled correctly, or suffered some sort of accidental abuse in transit, but whatever it was, the machine is not functioning like it should.

We called Lenovo back. Their solution was to try to schedule a replacement of the motherboard. I said ‘no dice’. Any machine with that much wrong with it can’t be trusted by us. We’d give it to a user and it would conk out somewhere 3,000 miles from the nearest Lenovo tech. I wanted to exchange the machine.

Of course, the tech on the phone said no. That’s what first level techs are for. Well, for that, and for handing me off to their supervisor. Who also told me no. I patiently explained the situation again, with the expectation that they’d just need to make a managerial decision to eat the potential cost in favor of placating an unhappy customer. But still, the answer was ‘no’.

Alright, time to attack this from another angle. I called my contact at CDW, who had previously mentioned that, should I get any flack from Lenovo, he could try to intercede on my behalf. I had hoped not to take advantage of the offer, but there was no way around it. I talked to Gordon, explained the story, and he was as incredulous as I was. He called their inside Lenovo guy who gave him the contact point at Lenovo corporate (or whoever), and Gordon called me once he had them on the phone. The guy at Lenovo listened to my story, sympathized with me, and told me that he would escalate my case. Finally! I could expect a call back that day or the next. Awesome!

Today, that call came, and I’ve got to say, it didn’t go as I planned. The escalation agent called Ryan first, since his name was attached to the original case. After he got off the phone, he had sort of a glazed look in his eyes as he swung around in his chair.

“You’re not going to believe this…”

Well, the story was so completely insane that, while I believed him, I still didn’t think it was true. I mean, there’s really no way. I had to verify something this stupid, so I had Ryan forward me the escalation agent’s contact info, and I called him myself. I patiently explained the situation, and he listened, matching me patient for patient. At the end of my speil, I said, “…so clearly, it makes sense for Lenovo to swap us machines now, rather than putting both of us through the hassle.” At which time, he repeated, verbatim, what Ryan had told me. My jaw dropped.

Because it’s not often you get an opportunity to observe utter stupidity in crystalline clarity, I wanted to preserve the moment. I said to the escalation agent, “I understand what you’re saying to me, even though I don’t agree that it should be necessary. What I need from you now, though, is the text of what you just told me, in an email. I need to see it in order to act on it. Can you send me that email?” And he agreed.

Ladies and gentlemen, I give you the email that the official Lenovo SRR Support agent sent me (emphasis mine):


As I discuss with you and Ryan, we have two options at this point–

1. I can submit request for the machine to be replaced but this would not qualify and I am absolutely sure the request would be decline due to lack of service repair history.

2. Write letter to CEO of IBM expressing your ordeal with this machine and why you believe it should be replaced. The letter will generate an executive complaint to the Lenovo Customer Relations. You should be contact no later than 2 days from when this has been submitted.


ThinkProducts SSR Support

Are you KIDDING me???
What is this, Bob’s Discount Laptops?

Over the phone, the rep actually walked me through typing in the url, and instructed me to click on the button to the left, marked “Send Email”.

I’m not beside myself with rage. I don’t care that much about a single laptop. I’m beside myself with confusion. I don’t even know what to think.

Please tell me, I’m not alone in thinking this is just bizarre, right?

Take my job…please!

So, in my last post, I let everyone know that I had given my work 2 months’ notice that I was leaving. So if you didn’t see that post, then, surprise! I’m leaving my job (though not the blog) after I find a successor.

Well, this post is the inevitable outcome of the previous post…

My company has a job opening for a senior system administrator. I’ve got ~2300 subscribers here, the vast majority of which are sysadmins, and a readership greater than that through distribution and syndication. I’m hoping that you, or maybe one of your friends, might be a good fit for this job, because I really want to leave my company in a good place, with someone who will take care of the infrastructure and the people behind it.

Here’s the job description:

We are seeking a SAGE Level IV Senior System Administrator with 5-8+ years of experience who is fluent in Linux, with several years’ experience leading a system administration team. The System Administrator, who will report to the COO, will immediately be responsible for:

  • Management of the IT infrastructure and the architecture, execution, and maintenance of the servers, networks, enterprise storage, and WAN links;
  • Management and maintenance of the production and backup server environments;
  • Management and mentoring of junior staff;
  • Support and creation of shell scripts, cron jobs, and other miscellaneous UNIX services;
  • Working with senior management to establish policies toward servers, desktops/laptops, and mobile devices;

Attention to detail, creativity, and ability to work as part of a team are fundamental attributes of everyone in the company. This position will be filled by the best candidate in either our Berkeley Heights, NJ or New York City offices. Travel between the offices will be required, as well as a willingness to work from either NJ or NYC. The successful candidate will:

  • Have 5 to 8+ years as a system administrator, including at least two as a senior system administrator (SAGE Level IV);
  • Have 5+ years using Linux in an enterprise settings, and 2+ years supporting Windows desktops;
  • Have experience supporting installations of DBMS such as PostgreSQL and/or Oracle (being a DBA is a plus, but not required);
  • Have an in-depth knowledge of backup practices and network troubleshooting;
  • Understand subnetting, SMTP, DNS, TCP/IP (familiarity with IPv6 is preferred), and IPsec-based VPN tunnels;
  • Have a very high to expert level knowledge of Linux;
  • Have the ability to travel to our colocation facilities in Carlstadt, NJ and Philadelphia, PA;
  • Be familiar with storage systems from the LUN level to the inode, have experience configuring LVM, and be comfortable working with SANs;
  • Have thorough familiarity with ESXi 4.x and legacy VMware Server;

Basically, I want to hire someone who is better than I am. My company is fun to work for, very successful, consistently growing, and currently has plans to expand. If you’re the right candidate, and you get this job, you’ll work with a bunch of really smart people who are determined to succeed. I consider it an honor to have been the sysadmin here for 6 years, and I will miss working here.

If you meet the above description of a senior system administrator, please drop me an email at [email protected] with your resume, because I’d love to talk to you about this.

The way I’m planning to go through candidates is pretty simple. I’ll read all of the resumes (or at least skim them), pull the ones I’m interested in, evaluate them pretty thoroughly, email to schedule a technical phone interview (which will last 15 minutes to an hour, depending on the candidate), then pull the cream of that particular crop in for physical interviews with myself (a technical-specific interview which will involve extensive use of the whiteboard), and other members of the company to evaluate the candidate’s ability to work with the team.

I got some practice when I was hiring for my junior admin a while back, so hopefully this will be smoother and quicker. We’ll see. I’m definitely open to advice, if any of you have experience hiring your successor. Please let me know in the comments!

Today, on a Very Special Standalone Sysadmin

So…over the past few days, I’ve told a lot of people about this, a lot of times, but for some reason, this seems to be the most difficult of the bunch…

This past Monday, I resigned my position as System Administrator at my company. My wife and I are moving back to Columbus, OH so that she can work from her home office, and we can be nearer to our friends and family.

As for me? Honestly, right now, I’m not sure. I’ve been a full time, 24×7 365 sysadmin for 6 years without too much of a break. That’s a long time. I don’t think I want to jump into anything too quickly.

Because I still care about my company a great deal, I don’t want anything to fall apart after I leave. I also want the company to continue to enjoy (at least) the level of service that I gave to them. For that reason, I’m staying on for a couple of months to help find my replacement.

This makes ‘right now’ a very interesting time for me. It’s been a long time since I’ve been so unsure about what life was going to be like. Since I let my company know, I’ve told some friends, and to my surprise, the single most frequent question people have asked has been whether I’m going to continue to write this blog.

I am a system administrator – I think like one, I act like one, and I have done nothing else for a long, long time. I’m not going to stop being interested in the same topics as you, and I’m not disappearing from the internet. I still feel like I have things to share, and I’m damned sure that I’m not out of opinions, so yes, I’m going to continue to blog here.

Actually, I’ve been thinking about spending a lot more time writing, once I unwind a bit. I enjoy it, I’ll have the time, and people seem to enjoy reading what I write, so maybe I’ll finally get to that whole book thing.

Until that time, though, I’ve got a couple of projects I need to finish at work. I want to leave the infrastructure in a better state than it is, so I’m going to be busy pushing through those changes.

As I mentioned, I’m also going to be looking for a replacement ‘me’ to wrangle this herd, so I’ll be posting a job description in the coming days.

I understand that you may have questions or concerns, so if that’s the case, drop me a comment or an email and I’ll try to alleviate them. Thanks!