Introduction to Enterprise Storage

December 14, 2009

This weekend, there was a question posted to serverfault that asked about popular mass storage option. You can find the actual question here, but the gist was that the person asking had large storage requirements (>10TB) and didn’t have much knowledge about enterprise storage options.

I’ve put some thought into writing an “Intro to Enterprise Storage” type post for a while, because I remember back when I got my first SAN, I was confused as to how it was all supposed to hang together. I had used Direct Attached Storage (DAS), but I didn’t know it by that name, and I had used Network Addressable Storage (NAS), though I didn’t know it by that name either. I slowly kludged together a working knowledge of how storage “works”, although I’m certainly still no expert.

Hopefully, I do have enough knowledge to help some other people who may just be starting out. To help the person asking the question, I wrote the following answer. It took enough time that I thought I could make it do double duty as a blog entry, too.

If I got something wrong or left something out, drop me a line in the comments.

---

There are three types of enterprise storage:

• DAS – Directly Attached Storage

This is storage that is either internal to the machine or is an external array of disks connected to the bus of the computer. Examples would include anything from an internal hard drive to an external USB drive to a 12 bay SCSI RAID array. The defining factor is that the storage is on the local bus.

• NAS – Network Addressable Storage

This is storage that is available over the (usually) TCP/IP network, using higher level protocols. If your desktop has an NFS mount, you can consider that data to be on a NAS server. NAS is a relative idea. The actual storage could be internal hard drives on another server or be a network appliance that does nothing but provide storage. Either way, from the perspective of the client, it’s network addressable. The protocols that are typically included as NAS are NFS, CIFS/Samba, and FTP, though there are probably others

• SAN – Storage Area Network

Network addressable storage used a TCP/IP network to transport data using higher level protocols. Storage Area Networks use lower level protocols to present block devices, and typically they their own network fabric to do it, though not necessarily. There are two very common SAN fabrics: iSCSI and Fibre channel. The traffic is sent to the fabric via a Host Bus Adapter (HBA), which you can think of as a SAN network card.

iSCSI utilizes its own IP network, and is capable of using standard network adapters and switches, which makes it relatively cheap. iSCSI HBAs are a lot like normal network cards, although many of them perform TCP offloading to conserve processor use on the server. Access control is provided through permitting or denying IP addresses from accessing disk resources. As with any other IP based network, ACLs, firewalls, and routing can be used as traffic control devices, although this is frowned upon by many due to the latency that it causes. Speeds range from 1Gb/s to 10Gb/s for brand new, highly expensive network gear.

Fibre Channel utilizes a completely separate mesh from existing IP based networks. It requires FC HBAs which are either native fiber transceivers or copper SFP connections. FC switches are available to provide additional network segments. Machines are addressed using World Wide IDs or Names (WWIDs), and access control is provided using these addresses by switches and the SAN storage itself. I’m not sure of the current maximum FC speed, but I know that 4Gb/s is available, and I suspect 8Gb/s is too, for enough money. If not, it will be soon.

The typical SAN storage is a large array of disks in an expensive, highly redundant enclosure. Using embedded software on the array, the SAN administrator uses the available disks to create slices, identified as Logical Unit Numbers (LUNs), which are presented to the specified servers as raw SCSI devices.

On the server itself, these devices are treated identically to any local storage. Partitions are created, filesystems are put in place, and data is written normally.

iSCSI and FC aren’t the only SAN fabrics, but they are the most common. Also available is the ultra fast (40-50Gb/s) ultra expensive (> $500 for an HBA) infiniband fabric, as well as the ultra cheap (off the shelf NICs) relatively slow ATA-over-Ethernet, which is sort of a layer-2 equivalent of iSCSI.

« Previous Entry


My Sysadvent Calendar Entry is up!

Next Entry »


Technorai Claim Code – Don’t Click Me!

Send this post to...





Posted in General