On July 31st an article titled iSCSI Killer was posted on Slashdot, which created quite a flurry of blog posts and activity. The Slashdot post references this linux journal article.
There has been a huge shift in storage technologies over the last year or two. It’s been very exciting, and there are a large number of very compelling vendors who offer some great products and services.
I’m not going to give a tutorial on networked storage, but ATA over Ethernet (AoE) is a lightweight low level network protocol which is NOT IP based. AoE can’t be routed since is isn’t IP based, and it gets punted over to all devices on the switch. The current mainstream, network based protocol is iSCSI. iSCSI shunts SCSI commands into TCP frames. iSCSI is a IP based protocol which may be routed.
Yes AoE seems like a lot of fun, and yes it is a different option than iSCSI, but I still say its crap. First off I’ve seen iSCSI work in high transaction environments with updates in bursts of 1Gb in a few minutes. The clients have no problem. The 3Par server had a TOE, but it was still a lot less expensive than the EMC/Veritas cluster we had. Screw 30% performance degradation from TCP/IP overhead, ’cause iSCSI with a TOE is still cheaper that what was available in 2003/2004.
iSCSI may be routed across switches which is a huge deal. Why, because for most networks I build out I have a few routers and lots of switches, often a switch for every-other rack, and two routers for each ingress/egress. What good it 10Tb of data, if 90% of the machines in the co-lo can get to it!?
As for a large storage array sitting behind a single host, something that is very useful in a small business for disk based backups, may be easily accomplished via iSCSI. Check out this article on building an iSCSI array. The article singles out open-e as the best of breed.
From my limited research it seems that open-e is a tweaked version of Debian (my favorite) running a flash based box with an IDE controller.
Note that most of the articles list a price point of $1/Gb. Amazon’s 3S service charges $0.15 per Gb per month, and $0.20Gb foreach Gb transferred. So you would reach $1/Gb after 6 months of 3S service, even without transferring data! Goes to show you many times it makes sense to grow your own.