Affecting Change or Knowledge Workers as a Political Class?

In my heart of hearts, I’m apolitical. In an ideal world, I just wouldn’t care about the political dealings on a national scale, and I used to look at shows like Meet the Press as the equivalent of E!TV – Sunday morning gossip shows talking about things that I don’t care about. It’s with a sense of irony that I realize that I treat politics in rather the same way that bad companies treat IT – as long as it’s working well enough that it doesn’t cause me major problems, I can pretend it’s not there.

But lately, politics has stopped being quite so “set it and forget it” for me. Not only are there the normal ever-present threats to my online world (like net neutrality), but in the past year or so, it has become apparent to everyone that conspiracy theorists were onto the right idea despite themselves, and that the US government is, in fact, into everything, as deeply and often as possible. (As an aside, if you haven’t heard about TAO yet, make sure to check it out and then re-consider your thoughts on the possibility of BadBIOS).

Anyway, political crap is leaking over into my world, and it’s annoying.

The part that makes it interesting to me is that the biggest revelation in all of this was from Ed Snowden, a system administrator. It’s a little bit like Hobbits in Lord of the Rings…no one knows we exist for years until suddenly the fate of the world hinges on the activities of one of us.

Other people are recognizing that we work in an important intersection of knowledge and responsibility, too. I came across a presentation from this year’s Chaos Communication Congress in Germany. It was a talk by Jacob Appelbaum and Julian Assange, who were introduced by Sarah Harrison. The name of the talk was SysAdmins of the World Unite.

Most of the talk is on YouTube, but there are some cutouts. The audio recording, in its entirety (at least as far as I can tell) is available on SoundCloud. To save you the time, I’ve transcribed some of the more interesting parts:

Sarah Harrison: Why are sysadmins playing an important role in this fight for freedom of information?
Jacob Appelbaum: All of us have agency, but some of us have more agency than others, in the sense that you have access to systems that give you access to information that help to found knowledge that you have in your own head. So someone like Manning or someone like Snowden, who has access to these documents in the course of their work, they will simply have a better understanding of what is actually happening. They have access to the primary source documents as part of their job.

This, I think, fundamentally, is a really critical, I would say a formative thing. When you start to read these original source documents, you start to understand the way that organizations actually think internally. This is one of the things that Julian Assange has said quite a lot. It’s that when you read the internal documents of an organization, that’s how they really think about a thing. This is different than a press release.

And people who have grown up on the internet, and they are essentially natives on the internet, and that’s all of us, I think, for the most part. It’s definitely me. That essentially forms a way of thinking about organizations where the official thing that they say is not interesting. You know that there is an agenda behind that, and you don’t necessarily know what that true agenda is.

And so people who grow up in this and see these documents, they realize the agency that they have. They understand it, they see that power, and they want to do something about it, in some cases. Some people do it in small starts and fits. So there are lots of sources for lots of newspapers that are inside of defense organizations or really really large companies, and they share this information.

In the case of Chelsea Manning and in the case of Snowden, they went big. And I presume that this is because of the scale of the wrongdoing they saw, in addition to the amount of agency that was provided by their access and by their understanding of the actual information that they were able to have in their possession.

Sarah Harrison: And do you think that its something to do with, being technical, they have a potential ability to find a way to do this safer than other people, perhaps? Or…

Jabob Appelbaum:
I mean, It’s clearly the case that this helps. There’s no question that understanding how to use those computer systems and being able to navigate them, that that is going to be a helpful skill. But i what it really is, is that these these are people who grew up in an era, and I myself am one of these people, where we grew up in an era where we were overloaded by information but we still were able to absorb a great deal of it. And we really are constantly going through this.

And if we look to the past, we see that it’s not just technical people, it’s actually people who have an analytical mind. So for example, Daniel Ellsberg, who is famous for the Ellsberg paradox. He was of course a very seriously embedded person in the US military, he was in the RAND corporation, he worked with McNamara, and during the Vietnam War, he had access to huge amount of information.

And it was the ability to analyze this information and to understand, in this case how the US government during the Vietnam War, was lying to the entire world. And it was the magnitude of those lies combined with the ability to prove that they were lies, that I believe, combined with this analytical skill, it was clear what the action might be, but it wasn’t clear what the outcome would be.

And with Ellsberg, the outcome was a very positive one. In fact, it’s the most positive outcome for any whistleblower so far that I know of in the history of the United States, and maybe even in the world.

What we see now with Snowden and what we’ve now seen with Chelsea Manning, is unfortunately a very different outcome, at least for Manning. So, this is also a hugely important point, which is that Ellsberg did this in the context of resistance against the Vietnam War. And when Ellsberg did this, there were huge support networks. There were gigantic things that split across all political spectrums of society.

And so it is the analytical framework that we find ourselves with still, but additionally with the internet. And so, every single person here, who works as a sysadmin, could you raise your hand?


You represent, and I’m sorry to steal Julian’s thunder, but he was using Skype, and um, well… [applause] … we all know Skype has interception and man in the middle problems, so I’m going to take advantage of that fact. You see, it’s not just the NSA.

Everyone that raised their hand, you should raise your hand again. If you work at a company where you think they might be involved in something that is a little bit scary, keep your hand up.


So here’s the deal. Everybody else in the room lacks the information that you probably have access to. And if you were to make a moral judgement; if you were to make an ethical consideration, about these things, it would be the case that, as a political class, you would be able to inform all of the other political classes in this room, all of the other people in this room, in a way that only you have the agency to do. And those who benefit from you never doing that are the other people that have that.

Those people are also members of other classes as well, and so the question is, if you were to unite as a political class, and we are to unite with you in that political class, we can see that there is a contextual way to view this through, uh, an historical lens, essentially, Which is to say that when the industrialized workers of the world decided that race and gender were not lines that we should split on, but instead we should look at workers and owners. Then we started to see real change in the way workers were treated and in the way that the world itself was organizing labor.

And this is a hugely important change during the industrial revolution. And we are going through a very similar time now with regard to information politics and with regard to the value of information in our information age.

I’m not sure I’m a big fan of framing our role as that of a political class. That being said, there really are some distinct parallels between today’s information worker and yesterday’s factory workers, and that I suppose you could look at this through a Marxist lens and see us as a modern day proletariat, but I’m vaguely uninterested in framing the discussion in that light. Someone else can hold that banner if that’s what they care about.

I’m much more concerned with is how we make decisions about what we should do with the responsibilities we have, and the knowledge we have, and the information that we have access to. The idea that we have a unique responsibility to society at large because of the privileged role we play in the modern workforce is an intriguing one. It’s not over-romanticizing to see that people in our positions, in the right circumstances, can make huge impacts – Edward Snowden is prime evidence of that. But every bit of leverage that we have which could be used for good can also be used for ill. And it’s not just black and white, either.

All of us only know what we know, and sometimes, we know what we don’t know. We never know what we don’t know we don’t know – the unknown unknowns, in other words.

If you are involved with military planning, or if you just play RTS games, you’re familiar with the concept of the fog of war, a lack of situational awareness because your information is incomplete. And so, decisions that you make because you believe them to be right may wind up being hamartia because your knowledge was incomplete.

That being said, every decision that we make, in our jobs and in our lives, can only be made with the cognizant knowledge that our information is lacking and incomplete. When it comes to making decisions that can dramatically affect others, though, that places an extra onus of correctness. Not only are we affecting ourselves, we’re affecting others, and in potentially unknown ways.

If someone were in a position to leak information that was extraordinarily damaging to an organization that was doing a perceived ill, is the wrongdoing enough justification by itself to publicly release that information? How much wrongdoing is enough? To employ some reductio ad absurdum, how much intentional littering does a company need to do before it turns into industrial waste?

And all of this presupposes that we’re essentially incidental non-combatants in this war on the public. If you answered the wrong job posting and found yourself as a sysadmin for Hank Scorpio, that’s one thing, but what if you suspected that a company were guilty of wrongdoing? If you go “undercover”, so to speak, as a sysadmin in that organization, in order to uncover wrongdoing, how is that inherently different than vigilantism?

And yet, that’s exactly what Julian Assange was promoting in the talk. Here’s the transcription:

The system that exists globally now is created by the interconnection of many individual systems, and we are all, or many of us, are part of administering that system, and have extraordinary power, in a way that is really an order of magnitude different to the power that industrial workers had back in the 20th century. And we can see that in the cases of the famous leaks, the wikileaks was done, or the recent Edward Snowden revelations, that it is possible now for even a single system administrator to have a very significant change to, or rather, apply a rather significant constraint, a constructive constraint, to the behavior of these organizations.

Not merely wrecking or disabling them, not merely going out on strikes to change policy, but rather shifting information from an information apartheid system which we are developing, from those with extraordinary power and extraordinary information into the knowledge commons, where it can be used to, not only as a disciplining force, but it can be used to construct and understand the new world that we’re entering into.

Now, Hayden, the former director of the CIA and NSA, is terrified of this. In Cypherpunks, we called for this directly last year, but to give you an interesting quote from Hayden, possibly following up on those words of mine and others:

We need to recruit from Snowden’s generation“, says Hayden. “We need to recruit from this group because they have the skills that we require. So the challenge is how to recruit this talent while also protecting ourselves from the small fraction of the population that has this romantic attachment to absolute transparency at all costs“. And that’s us, right?

So, what we need to do is, spread that message and go into all of those organizations. In fact, deal with them. I’m not saying don’t join the CIA. No. Go and join the CIA. Go in there. Go in the ballpark and get the ball and bring it out. With the understanding, with the paranoia, that all those organizations will be infiltrated by this generation, by an ideology that is spread across the internet. And every young person is educated on the internet.

There will be no person that has not been exposed to this ideology of transparency and understanding of wanting to keep the internet which we were born into, free. This is the last free generation. The coming together of systems of government, the new information apartheid, the linking together, is such that none of us will be able to escape it in just a decade. Our identities will be coupled to it, the information sharing such that none of us will be able to escape it.

We are all becoming part of the state, whether we like it or not, so our only hope is to determine what sort of state it is that we are going to become part of, and we can do that by looking, and being inspired by some of the actions that produce human rights and free education, and so on. Why people recognizing that they were part of the state, recognizing that their own power, and taking concrete and robust action to make sure they lived in the sort of society that they wanted to, and not in a hellhole dystopia.

While I can’t agree with a lot of what he says, there are some good points. The continuing digitization of our live is inevitable – there’s no going back from that. And I believe that we absolutely do need to work to determine the future of the world that we live in. It’s this gradual realization of mine – that it’s finally dawning on me, I’m just not able to blindly ignore the politics of the situation with my head in the sand. Things do seem to be flying off the rails.

So here’s where I put it to you, my readers. What do you make of this? Do you see any credence to the view that information workers are, in essence, their own political class (or do you see value in that)? I don’t want to live in a “hellhole dystopia” either, but I’m not yet convinced that becoming an active combatant in the battle for “knowledge commons” is the right way to go about it. Do you think it’s possible to affect change inside the system, as I hope it is?

Please, comment below and share your thoughts.

Cisco Switch-Profile Issues

So, whenever you’ve got a Fabric Extender attached to more than one switch, you need to configure the shared ports in both places. To make life “easier”, Cisco has the concept of “switch profiles”, where the two switches will have what amounts to a template, that gets applied. That way you don’t have to worry about doing the same thing in each place. Theoretically, it’s an awesome idea. In practice, though, I’m not even running the things in production and it’s causing me all kinds of problems.

Here’s an example:

core01# config sync
Enter configuration commands, one per line. End with CNTL/Z.
core01(config-sync)# switch-profile core-shared
Switch-Profile started, Profile ID is 1
core01(config-sync-sp)# int po1
core01(config-sync-sp-if)# switchport trunk allowed vlan 1-314,1050-1065
Error: This PC has already been configured outside switch-profile. Please configure further commands outside switch-profile or import the port-channel within switch-profile

So, lets check on it.

core01# sh run int po1

!Command: show running-config interface port-channel1
!Time: Fri Dec 27 19:55:43 2013

version 6.0(2)N1(2)

interface port-channel1
switchport mode trunk
switchport trunk allowed vlan 1050-1065
spanning-tree port type network
vpc peer-link

Alright. And in the switch-profile?

core01# sh run switch-profile | section .*interface.port-channel1$
interface port-channel1
switchport mode trunk
switchport trunk allowed vlan 1050-1065
vpc peer-link

Alright, yes, As you can see in the running-config, the ‘spanning-tree port type network’ line exists, but not in the switch-profile. So, now we have to get it out of there. The obvious answer seems to be to go into ‘configure terminal’ mode (now just ‘configure’ in NX-OS) and take it out. Lets do that:

core01(config-sync-sp-if)# buffer-delete all
core01(config-sync-sp)# config t
core01(config)# int po1
core01(config-if)# switchport trunk allowed vlan 1-314,1050-1065
Error: Command is not mutually exclusive

So we can’t update the switch-profile because it’s got configuration outside of the profile, and we can’t update it outside of the profile, because it’s managed by the profile? Ooooookay. Maybe we can import it.

core01# config sync
Enter configuration commands, one per line. End with CNTL/Z.
core01(config-sync)# switch-profile core-shared
Switch-Profile started, Profile ID is 1
core01(config-sync-sp)# import interface po1
core01(config-sync-sp-import)# verify
Failed: Verify Failed

Well, huh.

core01(config-sync-sp-import)# sh switch-profile status

switch-profile : core-shared

Start-time: 840920 usecs after Fri Dec 27 20:06:57 2013
End-time: 59573 usecs after Fri Dec 27 20:06:59 2013

Profile-Revision: 86
Session-type: Import-Verify
Session-subtype: -
Peer-triggered: No
Profile-status: Verify Failed

Local information:
Status: Verify Success

Peer information:
Sync-status: In sync
Status: Verify Failure
Following commands failed mutual-exclusion checks:
interface port-channel1
spanning-tree port type network

Alright, so the local switch successfully verified, but the remote switch failed. Lets check over there.

core02# sh run int po1

!Command: show running-config interface port-channel1
!Time: Fri Dec 27 20:09:10 2013

version 6.0(2)N1(2)

interface port-channel1
switchport mode trunk
switchport trunk allowed vlan 1050-1065
spanning-tree port type network
vpc peer-link

core02# sh run switch-profile | section .*interface.port-channel1$
interface port-channel1
switchport mode trunk
switchport trunk allowed vlan 1050-1065
vpc peer-link

Yeah, that’s pretty much exactly what core01 said, too.

Things like this are why I want to drink.

sigh. I’m going to be calling the TAC. Again, for the second time this week.


Alright, after talking with Cisco’s Technical Assistance Center (TAC), here’s what I’ve found out.

  • The answer to almost all of the inconsistency problems I’ve seen regarding switch profiles is to break the peer-sync (using no sync-peers destination <destination>), make each of the changes locally so that each switch is locally coherent between its local config and the switch-profile config, make sure that the two switches’ switch-profiles are 100% completely and absolutely identical, and then set up the peer-sync again just as it was before.
  • Although the guy on the phone said that there’s no document stating this, they have found that most of the problems related to switch-profile syncing is on non-FEX interfaces. That is, interfaces that are ‘local’ to the Switch itself, or a port channel. He recommended that I only use switch-profiles for those interfaces that are actually shared between the switches because they live on a dual-homed FEX.

There’s no way to remove an interface from a switch-profile, either. The only solution is to blow away the profile and re-create. As much…uhh…fun, as that sounds, it’s not something that I’d like to be doing a lot of. Anyway, hopefully this can help someone else out. Good luck, and let me know if you come across anything weird with switch-profiles, too. I’m interested to hear how many people are using them versus manually configuring each Nexus for each change you make. I’m SUPER interested if you use another tool to automate shared port configs on multiple switches. Comment below!

Google OAuth2 API – Howto (mostly)

Google has an amazing amount of data, and pretty much every project I’m involved in outside of my day job uses Google Drive (nee Docs) and Google Spreadsheets to store immense amounts of information. Looking at a spreadsheet isn’t necessarily the best way to take in information, so it’s useful to get that data out of it and see it in a way that makes sense to you.

For that reason, I’ve been writing an environment to rapidly spin up new apps that use the Google Spreadsheet API, and I put it on github.  The code I’m writing is called BootstrapMatrix because it takes a Twitter Bootstrap environment and adds an array of APIs, CSS and javascript libraries on top. It’s totally not done yet, but it’ll get there.

As an ironic twist, today, my friend (and fellow LOPSA Board member) Ski Kacoroski asked the following question on the LOPSA Discuss list:

We have a home grown account management app that was using the Google OAuth1 api to manage accounts in our Google Apps domain.  They have now changed to OAuth2 and things are breaking.  I have been trading emails with Google Support for over a week now with no luck in figuring out how to get a service account for Google OAuth2 working.  The support person keeps directing me to documents that do not match at all with what I see on the Google Console or other screens they direct me to and they refuse to start a webex to simply resolve the problem.

So, has anyone here written an app that uses the service account OAuth2 to connect to a Google API.  If so, I would really appreciate a bit of your time as I am pulling out what little hair I have left over this.



I’m new to the whole Google OAuth API thing, but I figured I could write what I’ve learned in the past week or so, plus it will help cement it in my brain.

I’m going to explain what I’ve found out as though it’s all brand new. Any explanation of things you already know is accidental, so please bear with me.

The Google API console that you use to manage your keys and credentials is here:

Any existing projects that you have there will show up. Click on it, and you’ll go to a blank page (at least mine is blank) with a menu on the left:

Click on “APIs & auth” and you’ll have subchoices:

Under APIs, all of the various Google services default to “Off”. Click the “Off” button to turn them on:

It may take some time, because backend work is being done to generate the necessary credentials for each service. Now, click Credentials on the menu to the left.

At this time, you have one of three choices, depending on how your application needs to access the resources. If you have specific resources that are shared with specific people (such as Google Docs or Google Drive files), then you want to use the OAuth settings at the top:

You can have multiple redirect URIs and javascript origins – they’re the places that an application which is using your key is allowed to point people when they sign in. You want to have this be the URL of your application.

It doesn’t sound like you’re using Public API access, which would be useful if you wanted to embed custom Google Maps on your website or something like that which isn’t tied to a particular user account.

The other important thing on that page is at the bottom, “Key for server applications”:

Although I haven’t seen it referred to by Google anywhere, the “API Key” listed there is what a lot of software and libraries call the “Developer Key”, as I make reference to in this file.

Once you have the developer key, the client ID, and the client secret, you’re ready to rock and roll.

The general workflow for logging in via OAuth is like this:

1) Instantiate a new instance of Google_Client.
Use the object methods to set the application name, client ID, client secret, redirectURI, and developer key.

2) Set the scopes of the GoogleClient object.

Scopes are a little bit crazy, imo. Google loves APIs. They have so many of them that there’s actually an API to query the APIs. You can read about it here:

In essence, what the scopes do is to say, “here are the permissions that my app needs”. On the back end, this is then compared with the APIs you signed up for above (if you didn’t request Google Maps API access, but your app is requesting it, things could go badly), and it presents those permissions to the user, as in “Is it okay if this app can use your credentials while you’re logged out?”.

To be honest, I still haven’t figured out exactly how to use the discovery API (maybe someone else can chime in and we can both learn). I used examples from various places, and ended up with code that works, though it’s very much cargo cult (I’ve only been doing this for a few days).

I’m doing this:
$GoogleScopes = array(

which pleases the Google Gods and gives me access to the spreadsheets I need, with the caveat that the user accessing them must have read/write access. I’m still not certain why. I am apparently not the only person experiencing this. A friend turned me on to this blog entry by the folks who write Basecamp.

OK, we’ll assume that your scopes are now set up, either through luck or magic, and move on to the next step:

3) Authenticate the user 
So, when your user shows up fresh and new, they’re not going to be authenticated with Google and they’re not going to have any session variables on your site (at least, that’s what I’m assuming). Your code will need to check to see if they do, in fact, have a session token (or whatever method your code uses of retaining state between page loads). Assuming that they do not have a session token, then you need to display the friendly “Log in via Google!” box( and make it a link to $GoogleClient->createAuthUrl().

Remember up above when we said, “Hey $GoogleClient, here’s all of our secret stuff”? All of that was so this link could point to somewhere at Google that would absolutely identify our App as the source of this login attempt. The user clicks on the link and gets taken to a Google page which talks about the app they visited, what permissions are being requested, and so on. They have the choice at this time to log in, and if they do, they’re sent to the RedirectURI, and appended to that URL is a ?code=XXXXXX string. That XXXXX is their token, and uniquely identifies them.

So in your code, you first need to check if there’s an HTTP GET variable named “code”. If so, then you call $GoogleClient->authenticate() (note that I don’t know exactly what magic this does. I suspect that it is, in fact, authenticating US with google, not the client).

After the authenticate() call, then you set the session token variable (in PHP’s case, $_SESSION[‘token’]) to $GoogleClient->getAccessToken(). This ensures that your user can be identified persistently. At this point, you want to redirect your user back to your app (because remember, they’re still at /your/app?code=XXXXXX). so inserting a Location header redirection seems to be the right way.

Back to the code that checks for a “code” GET variable. We already saw what happens when ‘code’ is set. When it isn’t set, then you need to check to see if they’ve got a ‘token’ session variable (or whatever you decided to call it). If they do, then you use that to set their Google access token using $GoogleClient->setAccessToken($tokenvariable).

At this point, all of the unique stuff for setting up the user’s authentication is done, and you’re ready to use the API calls to access Google resources.
I know that this isn’t a complete guide, but I’m still learning, and honestly, some of it is still outside of my understanding. I hope you got something out of this, though, and maybe BootstrapMatrix can do something useful for you.