Well, this is embarrassing...

Date November 30, 2012

I found out today that I have been configuring Apache wrong for a while. How long?

Around 15 years. 

ಠ_ಠ
Everyone who has configured Apache has probably done work with the <VirtualHost> declaration. You know how it usually looks:

NameVirtualHost *:80

<VirtualHost myserver.whatever.com:80>
# apache stuff goes here
</VirtualHost>

But that's wrong. It'll work, of course, but it's wrong.

Continuing, if you have multiple sites, and you were devoted to doing it wrong (or at least, never bothering to really grok the documentation), you'd do this:

NameVirtualHost myhost.mydomain.com:80
NameVirtualHost myotherhost.mydomain.com:80
# and maybe we'll get fancy with some https
NameVirtualHost securehost.mydomain.com:443

<VirtualHost myhost.mydomain.com:80>
ServerName myhost.mydomain.com
# Other Stuff
</VirtualHost>

<VirtualHost myotherhost.mydomain.com:80>
ServerName myotherhost.mydomain.com
# Other Stuff
</VirtualHost>

<VirtualHost securehost.mydomain.com:443>
# ServerName securehost.mydomain.com
SSLEngine On
# Other SSL stuff here
# Other domain stuff here
</VirtualHost>

Yeah, that's wrong too. When you do it, you get these niggling little errors:

[warn] NameVirtualHost myotherhost.mydomain.com:80 has no VirtualHosts
[warn] NameVirtualHost securehost.mydomain.com:443 has no VirtualHosts

...but it works. So you're like, "that's weird. well, it works, I'll figure it out later", but you never do.

Well today, I did. And I learned something. And I'm going to tell you.

See, my misunderstanding was all in the "NameVirtualHost" line. I thought, because it's called "NameVirtualHost", it wanted the name of a virtual host. Call me crazy.

Instead, what that ACTUALLY does is point to an IP address that Apache is going to listen on. It's just that by putting the domain name, it looks it up, and comes up with the IP address.

So if myhost.mydomain.com has a DNS record (or a host file entry) of 192.168.1.10, then these two lines are equivalent:

NameVirtualHost myhost.mydomain.com:80
NameVirtualHost 192.168.1.10:80

...and that's why it always works. Up above, when you wanted to enable ssl, you were like, "I've already go a web server; let's make this happen!", so you just added a CNAME (or maybe another A record) for securehost that matched the IP of myhost, and went on your merry way.

The other symptom of this problem, apart from the "has no VirtualHosts" error, is whenever you have multiple SSL-enabled sites on the same IP address. The way https works is that the browser connects to port 443, magic happens when the certificates are exchanged, and then your web browser is like, "it would be a shame to waste this secure tunnel. How about you give me securehost.mydomain.com?".

If you have "othersecurehost.mydomain.com", too, then you might see the problem. The certificate has already been exchanged by the time the server figures out what site the client wants...and the way it works in the apache config is that the first virtualhost encountered take precedence, so your browser might WANT othersecurehost, but the certificate it gets will be for securehost.

This sort of things will be obvious if you administer multiple externally-visited https-enabled sites, because you'll get certificate warnings all over the place. But 99% of what I work with has been building https-enabled internal sites where I just never, as a rule, bothered to spin up a good reliable certificate service and sign my own stuff, so I was always USED to certificate errors.

The site that finally helped me figure out what was going on is Apache Common Misconfigurations. And incidentally, there has been an extension to the SSL standard that allows multiple SSL-enabled sites to be hosted on the same IP address. Not every browser supports it (for those of you who still cater to IE6 users, you would now have even more sympathy from me, if that were possible), but if you have a reasonable expectation that your visitors' web browsers were written after the Bush administration, then you should be OK.

So anyway, today I'm happy to admit my blinding ignorance if you will accept for it an offering of knowledge. I apologize for going this long without investigating that one weird little error that didn't seem to be causing problems. I'll try not to let it happen again. Too often.

  • http://peelman.us Nick

    NameVirtualHost should win the award for worst Apache directive name ever. I'm sure that on some level, it makes sense. But for almost every common use case, it does not.

  • http://www.standalone-sysadmin.com Matt Simmons

    Nick: Completely agreed. I'm sure that at some point 15 years ago or so, I looked at a sample config, said, "oh, that's obvious", and never looked back.

  • Derek

    I'm not sure the first example (*:80) is actually *wrong*. It may be doing something more than you specifically want (setting up NameVirtualHost on all interfaces on port 80), but that's not inherently *wrong*.

  • http://peelman.us Nick

    @Derek: the problem isn't with the NameVirtualHost directive, its with the VirtualHost directive below it. NVH and VH directives should match. If you use *:80 in the NVH directive, you should use *:80 in the VH directive. You provide the FQDN in the ServerName directive inside the VH block.

    OH, and Matt, you're captcha is killing me.

  • Derek

    Nick: Ah, yeah that I can see.

  • Joe

    So if I am reading this correctly, I should be configuring my server's IP address in the Apache configuration instead of using * in NameVirtualHost? That seems counter-intuitive, in that now if I clone the server, I have to take additional steps to tell software (Apache) about the new IP for the cloned server instead of just setting up the new IP on the network interface at the OS layer?

    From an enterprise sys admin management perspective, this adds complexity to an otherwise simple task. Or am I missing something here (fully acknowledging that I haven't read common misconfigurations link yet)?

  • Derek

    Joe:

    You could just do, as Nick says, always use *:80 everywhere. That would be syntactically valid.

  • Joe

    Derek: That's what I thought (and what I currently do today across my server farms), but was getting the sense from this article that while it technically works, it's not the "right way" of configuring virtual hosts.

    I just think that software should be IP agnostic wherever possible (meaning it's the OS responsibility to handle that). There are situations where you may have multiple interfaces or IPs and you want a service only listening on one of them, in which case you usually have to make the software aware of IPs to know which ones to bind to or use a firewall/iptables.

  • http://www.standalone-sysadmin.com Matt Simmons

    Joe: As long as you want every VirtualHost to be appearing on every IP address, then yes, *:80 works everywhere.

    And Nick: the math thing? Yeah...but it has all but stopped spam completely. I think I get one spam comment every few months or so, whereas I was getting deluged by it.

  • http://buckinghaminquirer.blogspot.com Ryan Miller

    Heh, yeah, this bites everybody, sooner or later, and if you use mod_vhost_alias things just get that much worse. Counter-intuitive isn't even the word for apache config files, but of course it does everything and it's pretty reliable, so we keep using it.

    But I hope people in general get more comfortable with PKI, because internal HTTPS without certs isn't that great, either, when you think of all the ways that somebody on your LAN can spoof ARP, muck with MAC addresses, and all sorts of other ways to MitM you if they want to listen to your traffic. And of course, if you don't care if internal users are listening, then why use HTTPS, just make your sites digest auth to keep your passwords locked up and don't worry about it.

  • http://www.standalone-sysadmin.com Matt Simmons

    Ryan: The biggest problem with a CA that I've seen is the administration of all the existing ones suck.

    I don't know why there isn't a much better interface than command line openssl or Microsoft's CA snap-in. Maybe there is and I'm missing it. Do you know of anything?

  • R. Denison

    So here's a potential use case for the NameVirtualHost directive "as-is". Assume a server with multiple networks and network IP addresses, eg SAN, MGMT, and SVC, in the local hosts definition you might include the following:

    192.168.0.1 mgmt-net-if mgmt-net-if.subdomain.domain.example
    172.16.0.1 san-net-if san-net-if.subdomain.domain.example
    10.0.0.1 svc-net-if svc-net-if.subdomain.domain.example

    And the httpd.conf file snippet as follows:

    NameVirtualHost svc-net-if:80

    ServerName vhost1.domain.example
    ...

    Might be marginally useful for a stand-alone configuration, but I believe it makes having a single httpd.conf more self documenting, and that it can be pushed out with no modification possible with the trade-off of having to mange the hosts file, or using dynamic DNS / DHCP in the server environment.

    Caveat Emptor: the org I work for currently does one-offs like there's no tomorrow, so this resides in the realm of theory, but...

  • Andrew

    Most people know about 'apache2ctl configtest' but not many people seem to know of 'apache2ctl -S'. This shows how Apache is parsing the virtual host config and is really useful.

  • http://buckinghaminquirer.blogspot.com Ryan Miller

    Matt: I use easy-rsa from the openvpn package; there's also TinyCA and a couple of other reasonable GUI frontends that don't require any grand server setup themselves. Pretty much anything is better than openssl cli or the snap-in, which are not only slow to use but also make it really easy to make mistakes. Whatever you do, don't use cacert as so many people erroneously suggest--major security flaw.

    Andrew: sheer genius, and proof that on any given day you can learn something new about Apache that will rock your world, no matter how many years you've been using it.

  • http://peelman.us Nick

    @Andrew: Agreed with Ryan. That is one of those damned commands/flags I wouldn't find myself in a million years, but as soon as somebody shows it to me, it turns that particular aspect of my world on end. Thanks!

  • http://satanspanda.com Matt

    I actually learned this exact same lesson.... 2 weeks ago. TBH it didn't make sense to me why doing things the way i did them resulted in (not) an error didn't make sense till i read this. thanks for the post

  • graham

    I think its better to use hostnames inatead of ip addresses in your config. This way, if you move your servers to different ip addresses - setup to production for example - you dont need to reconfigure apache. Just change the dns it looks at - which you have to do anyway - and all is good.

  • Nathan

    The biggest hurdle with SNI right now is Windows XP doesn't support it on any browser even the more recent ones, and unfortunately there is still a lot of XP out there.

  • Pingback: IT Automation Digest for December 5, 2012 | Puppet Labs

  • Pingback: Common Apache Misconfigurations | Sam Howard - IT Architect