Intermediate Certificate Hell

Date September 10, 2010

I've got a small infrastructure. I think, all told, I deal with something like 3 or 4 SSL certificates, maximum. Probably less. This means that I don't deal with them very often, and last night was my first time dealing with intermediate certificates. Ye gods.

A quick introduction to how SSL certificates work may be in order...

First, SSL certificates deal with public key cryptography and key pairs. There is a private key that remains protected, and is used to create a public "certificate", and can be used to "sign" (in other words, vouch for) other certificates.

Secure Socket Layers is all about trust. Making sure that the website you're visiting is REALLY the website you want to visit is important on lots of occasions. The way SSL does this is by having an organization that you trust vouch for an organization that you're not sure about.

Your web browser comes out of the box with several trusted certificates. These are the public certificates from Signing Authorities (SAs) that are trusted because...well, usually because they've spent a lot of money to put themselves in a position where they're trusted.

The benefit of having these pre-existing trusted certificates means that whenever the private key owned by the trusted SA signs someone else's certificate, your browser trusts that certificate, too.

If I, as standalone-sysadmin.com, created a certificate and presented it to your browser, your browser wouldn't trust it. Why should it? Anyone can generate their own certificate. In order to have your browser trust my certificate, I have to get it signed by a CA that your browser trusts. Once that happens, then your browser will see "oh, this is his certificate, and it's vouched for by someone I know, so I'm going to trust it".

Back in the good old days (or at least before a year or two ago), Certificate Authorities (CAs) just used their root key to sign certs. This had the benefit of simplicity...the same key that generated the public root certificate pre-installed into your browser also signed the certificate being presented to it. Then things got more complex...

See, the benefit of root key signing was also the risk. Since a root key was trusted implicitly, if that key was compromised, it could be used to sign ANY certificate, and malicious people could present seemingly identical sites to unsuspecting people. It's sort of like the equivalent of stealing the printing press and paper to counterfeit money.

The most likely way that the root key would become compromised is during the signing process. Since every certificate needed signed by the root key, the root key needed to be everywhere signing took place. The volume of certificates that get churned out by some of these CAs is huge, which meant that one hole in the system anywhere could be exploited to cause the whole system to come crashing down.

In order to avoid this problem, what CAs have implemented are "intermediate certificates", which alleviate the necessity of keeping the root key around by creating "certificate chains". The root key still creates the public certificate installed in your browser, but it no longer signs the certificates presented by web servers. Instead, the signed certificates refer to the intermediate certificate, and the intermediate certificate refers to the root key.

This picture from IBM does it much more justice than I can:

The problem for us is that the browsers don't inherently know anything about the certificate chain...so we have to configure our web servers to present the entire chain of certificates. To facilitate this, lots of CAs have provided ready-made certificate chains in .pem files. I use Thawte, and you can get their intermediate bundles from here. Essentially, it's just a text file:

$ cat SSL_PrimaryCA.pem 
-----BEGIN CERTIFICATE-----
MIIERTCCA66gAwIBAgIQM2VQCHmtc+IwueAdDX+skTANBgkqhkiG9w0BAQUFADCB
zjELMAkGA1UEBhMCWkExFTATBgNVBAgTDFdlc3Rlcm4gQ2FwZTESMBAGA1UEBxMJ
Q2FwZSBUb3duMR0wGwYDVQQKExRUaGF3dGUgQ29uc3VsdGluZyBjYzEoMCYGA1UE
CxMfQ2VydGlmaWNhdGlvbiBTZXJ2aWNlcyBEaXZpc2lvbjEhMB8GA1UEAxMYVGhh
d3RlIFByZW1pdW0gU2VydmVyIENBMSgwJgYJKoZIhvcNAQkBFhlwcmVtaXVtLXNl
cnZlckB0aGF3dGUuY29tMB4XDTA2MTExNzAwMDAwMFoXDTIwMTIzMDIzNTk1OVow
gakxCzAJBgNVBAYTAlVTMRUwEwYDVQQKEwx0aGF3dGUsIEluYy4xKDAmBgNVBAsT
H0NlcnRpZmljYXRpb24gU2VydmljZXMgRGl2aXNpb24xODA2BgNVBAsTLyhjKSAy
MDA2IHRoYXd0ZSwgSW5jLiAtIEZvciBhdXRob3JpemVkIHVzZSBvbmx5MR8wHQYD
VQQDExZ0aGF3dGUgUHJpbWFyeSBSb290IENBMIIBIjANBgkqhkiG9w0BAQEFAAOC
AQ8AMIIBCgKCAQEArKDw+4BZ1JzHpM+doVlzCRBFDA0sbmjxbFtIaElZN/wLMxnC
d3/MEC2VNBzm600JpxzSuMmXNgK3idQkXwbAzESUlI0CYm/rWt0RjSiaXISQEHoN
vXRmL2o4oOLVVETrHQefB7pv7un9Tgsp9T6EoAHxnKv4HH6JpOih2HFlDaNRe+68
0iJgDblbnd+6/FFbC6+Ysuku6QToYofeK8jXTsFMZB7dz4dYukpPymgHHRydSsbV
L5HMfHFyHMXAZ+sy/cmSXJTahcCbv1N9Kwn0jJ2RH5dqUsveCTakd9h7h1BE1T5u
KWn7OUkmHgmlgHtALevoJ4XJ/mH9fuZ8lx3VnQIDAQABo4HCMIG/MA8GA1UdEwEB
/wQFMAMBAf8wOwYDVR0gBDQwMjAwBgRVHSAAMCgwJgYIKwYBBQUHAgEWGmh0dHBz
Oi8vd3d3LnRoYXd0ZS5jb20vY3BzMA4GA1UdDwEB/wQEAwIBBjAdBgNVHQ4EFgQU
e1tFz6/Oy3r9MZIaarbzRutXSFAwQAYDVR0fBDkwNzA1oDOgMYYvaHR0cDovL2Ny
bC50aGF3dGUuY29tL1RoYXd0ZVByZW1pdW1TZXJ2ZXJDQS5jcmwwDQYJKoZIhvcN
AQEFBQADgYEAhKhMyT4qvJrizI8LsiV3xGGJiWNa1KMVQNT7Xj+0Q+pjFytrmXSe
Cajd1FYVLnp5MV9jllMbNNkV6k9tcMq+9oKp7dqFd8x2HGqBCiHYQZl/Xi6Cweiq
95OBBaqStB+3msAHF/XLxrRMDtdW3HEgdDjWdMbWj2uvi42gbCkLYeA=
-----END CERTIFICATE-----

This is where I started having a problem. See, I was looking at documentation that was telling me to use the SSL_CA_Bundle.pem file, which includes both the Primary Intermediate CA cert and the Secondary Intermediate CA cert. This was making my web server (actually a load balancer) puke.

The key? Only use the Secondary CA cert. I didn't even figure it out. I went to bed in disgust. My boss Andrew played with it after I gave up and finally figured that out. How we were supposed to know? I've got no idea.

Also, why we had to use the Secondary CA and not the Primary CA? I don't know that either. I examined my certificate with

openssl x509 -in mycert.pem -noout -text

and there's no indication of the secondary CA being used, so I've got no idea. If you do know, I'd love to hear an explanation.

So yeah, I worked on this for a total of 5 or so hours yesterday, all told. It was my own personal hell, but the certificate is installed, we've learned how to deal with this in the future, and I don't have to deal with this particular cert again until Sept 13th, 2013.

  • Pingback: Tweets that mention Intermediate Certificate Hell | Standalone Sysadmin -- Topsy.com()

  • Antiphase

    That "bundle" you've got above (SSL_PrimaryCA.pem) is just one cert. A bundle is just a file with multiple certs in it, one after the other; nothing special. What joins them together is the Issuer field, which is the same as the Owner (aka DN) of the signing CA cert, and the fact that relationship can be cryptographically verified.

    All that matters in the end is that a valid chain can be made from certificates presented to and known by the client

    Your load-balancers must either be a bit nuts, or the certificate (or certificate chain) you fed it were syntactically invalid or contained blank lines or dodgy characters or suchlike. I guess it should just serve up the file to the client to do with as it will.

  • http://www.paulgraydon.co.uk Twirrim

    I'm going through this lovely slice of hell right now. Yesterday our cert expired and whilst I'd managed to successfully change it for Apache, Zimbra, openldap and our tomcat apps, I've been having fun getting qmail to be happy with the cert.
    Courier-imap took me a few to figure out but got it sussed fairly quickly.

  • Mike

    Ah yes, SSL certs and their various intermediate bundles/etc are a particular fun hell, as I've found out over the past while. Figuring out which intermediate cert or chain bundle is appropriate for the type of SSL cert you have is probably the hardest part - I have to document & bookmark the download sites of the two SSL vendors we use because it seems I can never remember which one is correct (we use VeriSign and EnTrust). I finally documented every step & every resource so that next year I can do it quicker; or if I need to provision a new web server I don't have to second-guess myself.

    I've found these two "SSL checkers" to be helpful to verify that I have the right intermediate certs in place:

    https://knowledge.verisign.com/support/ssl-certificates-support/index?page=content&id=AR1130

    http://www.sslshopper.com/ssl-checker.html

    (Thawte has one as well)

    And as you touched on earlier, using openssl to examine cert and key files helps troubleshoot:

    openssl x509 -noout -text –in
    openssl rsa -noout -text –in

    For instance, you can use openssl to verify the modulus of the certs and the key files match each other (not the issue you had, but it was a weird issue we had on a server).

    Related to this, I have our monitoring app (Xymon) check for SSL cert expiration dates and alert when there's 14 days or less left on the cert, as that gives me time to renew it, schedule an outage, deal with any SSL vendor craziness (like updated intermediate chain bundles or moving to 2048-bit encryption, for example!), and then install & test the new certs. There are also scripts out there you can use to help monitor SSL expirations, such as:

    http://prefetch.net/articles/checkcertificate.html

    Regards,
    Mike

  • http://flickr.com/photos/shandrew/ Andrew Shieh

    If you're missing part of the chain, most browsers can go and download the appropriate intermediate certs via some of the fields encoded in the primary cert. However, this degrades performance, and different browsers behave differently.

    "If I, as standalone-sysadmin.com, created a certificate and presented it to your browser, your browser wouldn’t trust it. Why should it?"

    It should trust it at least as much as it trusts regular non-SSL sites. For many situations, encryption is more important than certification. Personally I would prefer having the certs treated like ssh host keys.

    Andrew (a new reader and a standalone sysadmin)

  • http://www.standalone-sysadmin.com Matt Simmons

    Thanks for the comment, Andrew. And I agree, treating certs as host keys (and complaining when they change) would be preferable to me, too. Unfortunately, the people running things don't see it that way.

    At least Firefox stores the certificate (after clicking 10 times) and will let you know if it changes.

  • Jason Hall

    What links the certificates together is primarily the Subject Key Identifier (SKI) and Authority Key Identifier (AKI) fields of the certificates. The lack of an AKI indicates that the "thawte Primary Root CA" certificate is a root CA. Comparing its SKI to the AKI on the "Thawte SSL CA" certificate shows that it is a subordinate (intermediate) of the root CA. Your certificate will then have its own AKI match up to the subordinate CA's SKI.

    Some other fields, like the Subject, may also be used to positively identify the chain but aren't relied upon like the key hashes of the SKI and AKI fields.

    I'm not sure why your device didn't like the root certificate, but it does make sense that you needed the subordinate certificate.

  • http://www.ssl247.com Extended Validation SSL

    I have been going through the same think with a wildcard cert on my server. I thought adding another website on a server would be easy, :( no it a bigger head ache then setting up the server in the first place.

  • Sean

    Instead of using openssl x509 layer, try the s_client layer. It will present the chain, and definitively test what the server presents as the intermediate chain. openssl has no internal certificate store to measure trust or validity beyond what the certificates themselves present.

    $ openssl s_client -showcerts -connect http://www.yoursite.com:443
    depth=2 /C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority
    verify return:1
    depth=1 /O=VeriSign Trust Network/OU=VeriSign, Inc./OU=VeriSign International Server CA - Class 3/OU=www.verisign.com/CPS Incorp.by Ref. LIABILITY LTD.(c)97 VeriSign
    verify return:1
    depth=0 /C=US/ST=YourState/L=YourCity/O=YourCompany/OU=Terms of use at http://www.verisign.com/rpa (c)05/CN=www.yoursite.com
    verify return:1

    A couple of days ago, the original version of the VeriSign International Server CA - Class 3 intermediate expired. We ran an even more puzzling issue. The newer version of of this Server CA - Class 3 intermediate has the same serial number as the previously issued, and now expired, version. So the transition in replacing is fairly seamless (which is nice), however the consequence is that my currently valid, and operational, SSL Certificate was renewed after VeriSign re-issued this Intermediate CA Certificate. The consequence is that the old Intermediate works fine with the more recently issued server certificate.

    To make matters worse, troubleshooting this with browsers is absolutely useless. Each browser/version seems to implement the validation of the Trust Chain differently. Some browsers do not pay any attention to the Intermediate Chain presented by the web server. Some browsers store the Intermediate Certificates locally, others "call home" asking for some remote authority (like Windows Update Server) to validate for the browser.

    Using the openssl command above was the only consistent method to determine the certificate validity...

    $ openssl s_client -showcerts -connect http://www.yoursite.com:443
    CONNECTED(00000003)
    depth=2 /C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority
    verify return:1
    depth=1 /O=VeriSign Trust Network/OU=VeriSign, Inc./OU=VeriSign International Server CA - Class 3/OU=www.verisign.com/CPS Incorp.by Ref. LIABILITY LTD.(c)97 VeriSign
    verify error:num=10:certificate has expired
    notAfter=Oct 24 23:59:59 2011 GMT
    verify return:0

    My own FireFox and IE installations had already both discovered VeriSign's new intermediate, I assume by the nature that I keep them both up to date. So when I visited my own site, I received no certificate validation issues and the site functioned properly.

  • Pingback: Fixing SSL Sadtimes (Intermediate Certificates and cPanel) | Ian Boswell()

  • ngowda

    Unable to configure the primary and secondary intermediate certificate Configuration in Apache