March 21, 2013
One of the many differences I've noticed with this infrastructure versus the others I've dealt with is that our database instances here aren't centralized at all. If a specific service (such as a website on a specific web server - we don't really centralize those either) needs a database, the database lives on that machine.
One argument is that the administrative overhead of separating services like this is excessive - each database, in our case, is listening on localhost so that external hosts can't connect, so a central host can't perform backups because it can't connect. This means that each machine needs to be responsible for backing up its own database. Basically, if you want to manage N database hosts, you need N instances of that management tool and information.
The opposing argument is that performance is decentralized - the entire infrastructure's database load is spread over a large number of spindles, so no one database can act as a noisy neighbor to the others. Also, failure is dispersed - if one database goes down / fills up / etc, it doesn't take anything else with it.
Personally, I argue for the former. I think that managing one database machine is easier and less error prone - there are fewer moving parts, and we can concentrate on making the few machines that do provide database services more robust. On the other hand, I certainly can't argue against the other side - it's very true that we're spreading the load over more spindles than I could afford to give any one (or two) machines, and to be certain, a failure of one centralized database would take down all services which require access to the machine.
When I was asked what other people did, I said that people tend to centralize, but upon reflection, I realize that I answered only based on my experiences. So who better to ask than you, dear reader. How do you arrange your database servers? Do you have one (or a few) centralized database servers, either standalone or clustered, or do you spread the load like we are currently? Please comment and let me know - I'm interested in hearing about what you do.