Brian Heaton of Government Technology Magazine reached out to me for an interview regarding the CPU Act currently making its way through the Senate. He quotes me throughout his article which went live today.
If you follow me on twitter, you might have seen an interesting occurrence on Friday…
I was whining.
My server is hosted at prgmr.com, and it’s honestly a pretty decent VPS. I’ve got 2 vCPUs, 1GB of memory, and it keeps up with my normal amount of traffic, which is to say somewhere in the ballpark of 250-1000 hits, which usually tasks the server with something like 2500 page views on the upper end.
All put together, my server handles most traffic spikes well, although I did have some consternation in September, when my BoothBabes post went viral. I got 50,000 pageviews in a single weekend, which was enough to crush the server.
On Friday of last week, the “50,000 page views” record didn’t just get broken; it got crushed. mangled. stomped on repeatedly. Friday was a “perfect storm” of “kill the server”. Here’s how it went down:
Thursday afternoon, Reddit user algrym submits my CPU Act post to /r/sysadmin. It eventually grows to a score of over 90 (which is good for that subreddit, and ensures that everyone who subscribes to /r/sysadmin will see it, which is around 11,500 people).
Nearly simultaneously, users grauenwolf and rsantoro submit the story to /r/programming and /r/technology, which have a combined subscriber count of almost 850,000. Together, the posts receive over 1,500 upvotes, ensuring that the posts go not just to the top of their respective pages, but also to the front page of Reddit itself, which sees God-only-knows how much traffic.
Apparently, someone liked it enough to submit it to Slashdot, where it got over a thousand comments, and on the front page as well.
If it goes on Slashdot and it’s interesting, you can bet that Fark will cover it as well, and sure enough, it hit the front page of Fark.
Somewhere in the mix, it also got covered by Ars Technica, along with being spread around Facebook by thousands of people.
So where does that leave us?
The sum total is pretty staggering. In terms of raw “hits”, on Friday alone, I got 51,025. Now, those are basically visits, and don’t count page views, or the number of actual requests made of a web server. In terms of page views, according to CloudFlare, I got 297,649 requests, which I have trouble fathoming. And if you ever wondered why you might want to use a CDN, here’s the statistic that sells it. Total requests, that is, for dynamic AND static content, was 4,638,389…and the best part was that CloudFlare handled 4,235,891 of those. Which is a damned good thing.
As it was, I got an average of more than one request per second on Friday, and at peak, it took over 20 seconds to render the page…so you can do the math there. There were a lot of requests that just didn’t get served, even with TotalCache.
On the advice of several twitterers (particularly @jpluscplusm), I actually took the blog down for a little while in order to try to set up Varnish, but I pulled the plug and rolled back changes after a half hour or so – serving a certain percentage of requests was better than serving no requests. Varnish is something that I will be implementing one of these weekends, though.
I also want to thank some other people who helped me out very much. The founder and current CEO of CloudFlare, Matthew Prince, went the extra mile in getting the engineers wrangled and more actively caching the page that was getting slammed.
I want to thank all of you who talked with me and offered your resources during the time. If I didn’t take you up on the offers, it was only because I didn’t want to make the service more unavailable during the transition than it already was – but know that I really do appreciate the offers and the concern. Thank you.
So, what could I have done better? Tons.
Firstly, Varnish will be put in place. I don’t know of a way to load-test against 500,000 requests (if you do, please comment!), but I can at least watch performance metrics and compare them to how the site scales now.
Speaking of, I’m going to start collecting performance metrics. Right now, the page loads entirely in 4.95 seconds. I know that because I have Firebug installed in Firefox, and I just checked it. That’s the kind of stat I should be monitoring and recording so I can watch it over time. So I’m going to start doing things like that, too.
Anyway, that’s enough post mortem for a blog server, I suppose. Thanks to all of my readers (old and new). Hopefully we’ll have that problem again in the future!
I promised an update with the information that I received from my survey of pay types, and here it is.
Over the past several days, I received 567 responses from people, both in the United States and around the world. The law that we were discussing was relevant only to the United States, so I’m going to give the stats with the international visitors at the end.
From the US, there were 489 responses. Here’s how they break out:
Hourly or Salary?
About what percent of the time do you receive any overtime pay?
(17) any employee who is a computer systems analyst, computer programmer, software engineer, or other similarly skilled worker, whose primary duty is—
(A) the application of systems analysis techniques and procedures, including consulting with users, to determine hardware, software, or system functional specifications;
(B) the design, development, documentation, analysis, creation, testing, or modification of computer systems or programs, including prototypes, based on and related to user or system design specifications;
(C) the design, documentation, testing, creation, or modification of computer programs related to machine operating systems; or
(D) a combination of duties described in subparagraphs (A), (B), and (C) the performance of which requires the same level of skills, and
who, in the case of an employee who is compensated on an hourly basis, is compensated at a rate of not less than $27.63 an hour.
And the law as the CPU act would have it (with the differences highlighted in red):
(17) any employee working in a computer or information technology occupation (including, but not limited to, work related to computers, information systems, components, networks, software, hardware, databases, security, internet, intranet, or websites) as an analyst, programmer, engineer, designer, developer, administrator, or other similarly skilled worker, whose primary duty is-
(A) the application of systems, network or database analysis techniques and procedures, including consulting with users, to determine or modify hardware, software, network, database, or system functional specifications;
(B) the design, development, documentation, analysis, creation, testing, securing, configuration, integration, debugging, modification of computer or information technology, or enabling continuity of systems and applications;
(C) directing the work of individuals performing duties described in subparagraph (A) or (B), including training such individuals or leading teams performing such duties; or
(D) a combination of duties described in subparagraphs (A), (B), and (C), the performance of which requires the same level of skill;
who is compensated at an hourly rate of not less than $27.63 an hour or who is paid on a salary basis at a salary level as set forth by the Department of Labor in part 541 of title 29, Code of Federal Regulations. An employee described in this paragraph shall be considered an employee in a professional capacity pursuant to paragraph (1)
It’s pretty obvious what most of these changes are doing, but a couple are pretty subtle.
Clearly, the authors of the bill feel like there are some…uhh…loopholes, in the current law which are letting some IT workers through the noose. They seem to particularly have it in for network and database administrators, although if you pay attention in paragraph C, even the trainers and team leads are drawn in.
Basically, anyone operates things in IT and works for wages and gets over $27.63/hour is going to be affected by this bill.
My initial response when I read this was “Why would anyone want to do this? Especially a lawmaker?” – Clearly the lawmakers aren’t in it for themselves since most of them barely know how to work their webcams well enough to proposition interns, which means something else must be causing this.
Fortunately, I didn’t have to do the digging. Someone else did it for me.
The sponsor is Kay Hagan. Listed in her Top 20 contributors are companies like Bank of America, Blue Cross/Blue Shield, and Time Warner. The cosponsors are Michael Bennet (Comcast, Qwest, DISH Network, Level 3, Time Warner), Michael Enzi (Blue Cross/Blue Shield, and John Isakson (Home Depot, Delta, AFLAC, Cox, Citigroup, & GE). So, you know, no one that would be interested in lowering their IT costs a bit. If anyone knows where I could get numbers based on what percentage of employees at those companies are wage versus salary, I’d like to see them.
So what do I think? I think that they want to eliminate the ability of IT workers to get overtime pay, regardless of how much we work. What do you think?