I have been researching storage for the past few days. I’ve been concentrating on iSCSI, since I was trying to keep costs down, and a fiber switch is pretty expensive (especially if I want to use it).
While researching, I chanced upon a technology I hadn’t heard before: ATA Over Ethernet (AoE). Unlike iSCSI, which transmits the data over TCP, AoE does it via layer 2 frames. This has the implication that, like Fibre Channel, it can’t be routed across different networks. For most people, this is not a problem. For some, it’s a deal breaker.
In the same way that iSCSI can use software initiators (which turn a computer into an “iscsi server”), there is software available to create AoE “servers”. This would be useful if you’ve got a large machine with many available disk slots.
There are also AoE arrays on the market. Coraid sells some very large arrays. They even offer a ready-made High Availability NAS gateway.
There are drawbacks, of course. There doesn’t appear to be a lot (any?) inherent security in the protocol. If anyone reading has experience, I’d be very thankful for some comments as to how host control is done.
I’ve read comments that were a few years old claiming that it wasn’t as stable as iSCSI, but they offered no evidence towards that conclusion, so I have no way of checking to see if their complaints have been resolved.
In the end, I still don’t know what I will do, but the more I read, the bigger a blip it is becoming on my radar. What do you think?
I read Jeff Hengesbach‘s blog every day, and I’m fairly sure that if you like my blog, you’ll love his. He recently started it, and every post so far has been really interesting. You should check it out.
Anyway, today Jeff wrote about iSCSI SANs. If you’re unfamiliar with the concept, here is a great introduction. You should also know that there are two competing standards, iSCSI (where the storage is accessed over standard ethernet) and Fibre Channel (where the storage is accessed over fiber optics cables). There’s some more in-depth explanation here, but it all boils down to speed and expense.
Among the in-the-know people (AKA: not me), it seems to be a sort of common knowledge that we’re going to be using iSCSI in the future, what with 10Gb ethernet being the standard just-around-the-corner. The current best-of-breed is 8Gb/s FibreChannel from IBM, but 10Gb would trump that, and the added bonus of using tried and true ethernet is very appealing to a lot of people.
Regardless of whether or not we end up with iSCSI I’m not convinced that copper will last us much longer in the grand scheme of things. I suspect that, maybe by the time 100Gb rolls around, we’ll be using ethernet-over-fiber for that stuff. The theoretical bandwidth of optical is just too high to ignore in the long-term. If you want to get all sci-fi, scientists have recently began working on entirely optical versions of most basic integrated circuits.
Just a random observation to spark discussion. What do you see happening in the future of SANs?
Browsing through Digg today, I chanced across this article about Google expanding their undersea cable portfolio. Aside from increasing the inter-Asian bandwidth by over 7Tb/s, it will be Google’s 2nd undersea cable, the first being the transPacific cable known as “Unity”. As recently as a year ago, the first cable was just a rumor. In February, they confirmed that it would be in place and operational by 2010. The first article I linked to also mentions that Google is looking at an African fiber to replace the aging SAT-3 cable. Offhand, to give you some idea of the magnitude of the new fiber, SAT-3’s bandwidth is a paltry 130Gb/s.
What does this have to do with small systems administration? Not much specifically, to be honest, but it’s interesting to me. I look at the global fiber network as a maybe a macrocosm of my own network. I look at the global fiber maps and note the critical junctures, the (sometimes lack of) redundant bandwidth in certain corners of the world.
If you’re interested in this as well, here are some interesting links that I found.
Eyeball-series.org has a couple of pages dedicated to mapping the exact spots where each of the trans-Atlantic and trans-Pacific cable landings. This is a very, very bandwidth intensive page. Far more interesting are the maps at the top of each page that shows the logical circuits. Here, I’ll save you time and bandwidth: Atlantic and Pacific.
This map from CNET gives you some idea of the intercontinental bandwidth available around the globe.
Here is a series of maps showing the history of undersea cables.
If you’re curious as to how many cables are out there, Wikipedia has a page dedicated to listing all the cables in the individual bodies of water. The main page on submarine communications cables has a great schematic of the composition of undersea cables. How cool does this look: