Another storage option

Date March 9, 2009

I have been researching storage for the past few days. I've been concentrating on iSCSI, since I was trying to keep costs down, and a fiber switch is pretty expensive (especially if I want to use it).

While researching, I chanced upon a technology I hadn't heard before: ATA Over Ethernet (AoE). Unlike iSCSI, which transmits the data over TCP, AoE does it via layer 2 frames. This has the implication that, like Fibre Channel, it can't be routed across different networks. For most people, this is not a problem. For some, it's a deal breaker.

In the same way that iSCSI can use software initiators (which turn a computer into an "iscsi server"), there is software available to create AoE "servers". This would be useful if you've got a large machine with many available disk slots.

There are also AoE arrays on the market. Coraid sells some very large arrays. They even offer a ready-made High Availability NAS gateway.

There are drawbacks, of course. There doesn't appear to be a lot (any?) inherent security in the protocol. If anyone reading has experience, I'd be very thankful for some comments as to how host control is done.

I've read comments that were a few years old claiming that it wasn't as stable as iSCSI, but they offered no evidence towards that conclusion, so I have no way of checking to see if their complaints have been resolved.

In the end, I still don't know what I will do, but the more I read, the bigger a blip it is becoming on my radar. What do you think?

  • Nick Anderson

    I don't think you can have a password for AoE devices. I think you might be able to restrict an export by mac address.

    Using AoE is pretty easy though.
    vbladed 0 0 ethX /dev/someblockdevice

    Then on the client side just modprobe aoe and run aoe-discover. You should get /dev/etherd/e0.0
    Notice the major and minor numbers (0 0). Those can be any numerical value to my knowledge. They represent the Shelf and Blade respectively (so you can keep track of whats what and where.)

  • Marc Cote

    We use three iSCSI Equallogic SAN's connected to numerous hosts using QLogic iSCSI hba's so we can get better reliability and throughput than using software iSCSI. Security can be based on IP address or chap username/password. When I need some extra storage on a physical server, I can just toss the software iSCSI initiator on there and connect to a LUN on the SAN.

  • JeffHengesbach

    Another one that is linux centric: Network Block Device. I've not used it so I can't really say much - but it is another option.

  • -dsr-

    AoE has no security, but since it isn't routable, physical security is usually attainable.

    iSCSI CHAP doesn't actually encrypt data in flight, so you need to use IPsec... which is frequently slow on the SAN appliance side.

    AoE is conceptually simpler, less complex to implement, and potentially faster. In exchange, you give up flexibility and features. That may be a win for you.

  • Jim B.

    I always liked AoE conceptually because I wanted the cheapness of ethernet without the overhead of tcp/ip, but other than coraid it completely failed to take off. FCoE on the other hand is the same basic idea and the industry seems to be converging on it.

  • Jason Antman

    I haven't looked into ATAoE in a year or so, but last time I did, I believe that security was physical layer only, as dsr said; anything on the physical layer has access.

  • Matt


    I don't think the security would be too big of a deal, but it's something to think about, always. I think in the way that I'd implement it, that broadcast domain would be dedicated to storage, with no configured IPs, at least that's how I'm imagining it in my head.

  • Christopher Cashell


    Considering the ubiquity and market penetration that iSCSI has, why in the world would anyone want to bother with FCoE?

    This has always struck me as one of the silliest waste-of-time ideas. There is essentially no advantage over iSCSI from a technical standpoint, and the standard for it isn't even finalized yet.

    Betting on a protocol that's not available and that offers no advantage over something that exists, is widely deployed, widely supported, and works just seems a little crazy to me.

    Of course, I'm looking at it from the perspective of a System Administrator. I suppose if I worked for a company that made Fiber Channel based hardware of some sort, and I saw huge chunks of my sales going to iSCSI storage, I'd probably try to come up with some way to stay relevant, too. I'd like to hope I'd come up with something less brain-damaged than FCoE, but who knows. ;-)

    Note, for the record, I currently make use of traditional Fiber Channel SANs and iSCSI. When I recently had to build a no downtime allowed Oracle RAC system, I built it on Fiber Channel. When I needed less expensive SAN storage for an Exchange cluster, I used iSCSI. Both have their place, for now. Heck, we even have an AoE based file share (built by another company we merged with) that seems to work well enough.

    Future decisions will be made based on the requirements involved. . . but I'm having trouble picturing any situation where I'd ever choose FCoE over traditional Fiber or iSCSI.

  • Ben

    I use Coraid's AoE in production. The general consensus is correct: Security is basically implemented via the physical layer. It does support MAC-based whitelisting on a per-LUN basis, but given how easy it is to spoof a MAC, that's not especially helpful.

    I just isolated all AoE traffic to a dedicated network.

    Overall I've been extremely happy with the product and the quality of the support. The engineers are extremely knowledgeable and helpful.

    The big win for us was that the 15 bay AoE chassis was similar in price to a typical 15 bay standalone RAID chassis, but with much more flexibility.