Storagezilla

Growing Iomega

Jerome Wendt has the opening segment of what looks to be a good interview with Iomega's Jon Huberman.

Here's the money shot for me:

Iomega is very different today than it was a year ago. A year ago, we were we were effectively a "Me-Too" product. We had a small team and no fundamental technology of our own. We were really just taking direct attached drives, adding a fairly vanilla Linux stack on top of it, a network chip set, and leveraging our channels and brand.

...

In just the last year, EMC spent about a billion and a half dollars on R&D to solve network and data storage problems. We now have the ability to leverage all of the appropriate technologies and investments that EMC has made and bring those to the small business in a manner that is easy to use and cost-effective for them.

There are a few technologies I'd like to see Iomega, and not EMC, ship as I think they're too important to price out of peoples range. 

With Iomega's area of the external storage market looking like it's the largest growth segment they can price at a level which an AX or an NX Unified Storage product weren't designed to reach.

That's going to be important going forward and in the near future I wouldn't be surprised if the next storage device some of you are evaluating for your home lab has the Iomega brand on it.

Iomega's low cost volume sales approach is something we're actively encouraging, this is in direct opposition to others who appear to be chasing after the small to zero growth high margin parts of the market and have priced themselves out of the low end entirely as a result of that.

Another reversal comes in the form of EMC's deeper channel relationship. 5 years ago a channel partner getting someone from EMC to even speak to them probably involved a minor miracle happening somewhere.

Yesterday we found out that CRN gave us the Gold in the Channel Champion awards with HP and IBM taking the Silver and Bronze in the category.

There's no crowing here. I know from personal observation how well HP & IBM take care of their channel partners and I tip my hat to them for showing others how it's done but what this tells us is that we've made a significant improvement to the way things were and we're starting to see the returns which can't be measured in revenue.

Returns like partner goodwill.

It's not perfect, there's more we can do for the channel and more we will do but we're going to keep working on what's now a critical component of our business and design products which they can use to solve their customers problems.

Applications at Hyperscale

With all the talk of Gmail going offline today you'd swear it was the first time it has ever happened. Here's something you might not have noticed, Google have been having trouble with Gmail for a long time now.

I can recall four or five times over the past 12 months where I've gone to login to the web interface and for some reason I couldn't get past the front page. You'd think this would be a case of fat fingering a password but a check of my desktop email client showed that it couldn't login either at that moment in time. Wait a few seconds to try again and it worked fine. Maybe it's my net connection you think and move on. Or perhaps Gmail burps every now and then and you pass it off as your net connection?

It is possible that we now have unrealistic expectations of availability coming from our "five nines you're paying me if you don't meet the service level agreement" data centre world. But what's Google's issue?

WE don't know for sure but I have a theory. Might it be Google's mythical infrastructure perhaps? Something, if you listen to the hype, every company should aspire to even if their infrastructure needs are relatively modest (if not fractional) in comparison.

Google's bluster around their infrastructure conveniently ignores that it's optimised for one hyperscale application. Their search application.

That's the cash cow and that's what they've tuned the hell out of their infrastructure for. Tuned the block size, tuned for the I/O patterns unique to that application and tuned for the read/write frequency of that workload. Everything which they can optimise to increase the performance of their search application means more money in the bank if they do so.

But then they come along and try and put Gmail on top of that and it's been stuttering away in the background ever since. It's also funny to note that Gmail, which one assumes is sharing -cough cough- Google's general purpose (Search optimised) infrastructure has issues and yet Google.com carries on without a blip.

The same way that Amazon.com coasts away just fine any time S3 is having a nervous breakdown.

Funny that.

Now, while Google suffers from trying to run things on an infrastructure optimised for one hyperscale application you could say that S3 suffers from the opposite problem by not being optimised for any workload at all. It's so general purpose that it's cheap to get started on but if you're very successful you'll quickly want to get off of it as you can't optimise any of it.

The only things which are not a commodities in infrastructures designed to support hyperscale applications are the performance optimisations.

Everything else you buy cheap and rack high.

Unified Storage: File System Deduplication

--Updated 7/7/09--

Having looked at this stuff on and off for a while now I find it's best not to be wedded to any particular data reduction technology.

To give an example of the pragmatism involved anytime I log on to the EMC corporate network various NAS (Network Attached Storage) file systems are mapped to my laptop. Ten or so of them in total and they're all vast oceans of unstructured data.

So when we looked at the biggest win for de-duplication in our own environment going after all this unstructured data in file systems was on the top of the list and it's sure to be up there for customers as well.

So to begin with, the question could be asked "Why compression?".

The simple answer is "It's the best technology approach for this kind of data".

Notice I said for this kind of data. I did not say it's the best technology full stop. This wasn't an exercise in favouring one approach over the other it was an examination of what could be done to decrease storage needs without hitting system performance.

The more elaborate answer is that not only is compression not memory intensive but it's an infinitely variable bit level intra-object deduplication process which offers the greatest savings for the oceans of unstructured data typically found in NAS devices.

That isn't an opinion it's a scientific observation. We used our own NAS file system data in our tests and I've inlined some of our results.

Technology	"Typical" Space Savings	Resource Footprint
File-level deduplication	10%	Low
Fixed block deduplication	20%	High
Variable block deduplication	28%	High
Compression	40% - 50%	Medium

Looking at our own data and taking them from the top we saw File Level Deduplication (AKA Single Instance Storage) provide some modest space savings but it was pretty light on system resources.

Fixed and Variable Block De-Dup come next both are resource intensive but variable block de-dup had the edge in space reduction. This is also shown by it's use in the de-duplication of backup data where the most benefit is seen in savesets containing repeated but block misaligned data. It's the same with Virtual Machines, when you think of it they're just like backups as if you put a bunch of them together containing the same OS image a variable level de-dup system will see a lot of repeated but block misaligned data.

Finally we come to Compression which showed the highest percentage of space savings for file system data, anywhere up to (and sometimes above 50%) and while the resources used are CPUcentric.

So if you look at those numbers scientific measurement showed us that for file system data compression provided 75% the cumulative space reduction of the four other approaches combined for file system data.

By combining intra-object de-duplication with file level de-duplication we found you can realise great space savings at a reasonable resource cost.

What happens when you need to read that data? Yes you need to decompress it but it isn't that cut and dried. De-compression happens in memory not on disk, so you read a smaller amount of data off of disk than you normally would which in turn offsets and sometimes eliminates the amount of time spent on de-compression. But it goes even deeper than that. It's an open secret at this stage that we're using the RecoverPoint compression algorithms. I mentioned in a previous post that RecoverPoint was a treasure trove of technology, it's compression algorithms are just one of the clever technologies in that product.

Celerra deduplication can decompress just the required data to service an IO so you don't even have to spend cycles decompressing the entire file if the entire file isn't what's needed.

So when I go to Celerra Manager and check the Data De-Duplication box under the NAS File System is that an all or nothing operation?

No it's not. The way the system is designed we have a policy engine which is always running in the background using tunable parameters to decide on what should be de-duplicated and what should be left alone. If it considers the file to be active or the file type has been singled out to not be de-duplicated it'll ignore the file. If the file meets the tunable criteria for being inactive it'll be de-duplicated. First the compression engine will check to see if there's any benefit to be gained from compressing the file, if there is it'll be compressed and single instanced, if there isn't it won't spend CPU cycles attempting to compress the file and it'll just be single instanced.

Drinking our own champagne on average across our systems we're seeing between 30% to 40% space savings with file system de-duplication turned on.

But the biggest win for me is that Celerra de-duplication is integrated with VBB for even higher speed backup and recovery. VBB is Celerra's block based high speed backup and recovery technology, now it'll backup and restore de-duplicated data which means you've less to write to and less to read from backup media when you take backups.

There's also full integration with Celerra Manager/Replicator/SnapSure/FileMover.

To conclude, as we could go into the guts of this forever, the first data reduction use case we went after was file systems and we found that compression with file level de-dup does a great job, but what about Databases or VMware or anything else?

Plugin architecture.

Use of one or more technologies does not preclude the use of any other technology or any combinations of technologies. Since we have a policy engine for orchestration we don't have to standardise on a single all or nothing approach.

I have no interest in a debate on the merits of one approach over another since at it's core it's a very simple question.

How much can you decrease your storage needs without impacting operational requirements?

And that's one of the questions driving our development of primary storage de-duplication.

My thanks to Celerra Engineering for supplying the technical details and figures contained in this post.

Unified Storage: FLR-C

As Unified Storage becomes a larger and larger part of EMC's storage business I've a few thoughts on the new Celerras but I'll admit that I'm interested in what makes my life easier so this will be in multiple posts.

Lets begin with FLR-C.

EMC introduced Celerra WORM (Write Once Read Many) capability into the Celerra Data Access in Real Time OS (DART) a couple of years ago. Off the top of my head I'd say four years ago but that's without looking it up. Celerra WORM evolved over subsequent generations into the current iteration of that functionality which is called Celerra FLR (File Level Retention).

FLR as the name states operates retention policies on an individual file basis meaning the files themselves cannot be deleted while subject to retention.

As of the latest release of DART, FLR now offers two different levels of functionality. FLR-E (Enterprise) and FLR-C (Compliance).

FLR-E I'd use for health and hygiene. If people aren't supposed to be deleting particular files from particular volumes as per business practice then you don't let them. But that doesn't stop a privileged admin from coming along and deleting the file system as part of day to day storage administration even if the admin can't remove individual files.

On the other hand FLR-C meets SEC Rule 17a-4(f) compliance requirements and locks things right way down. Not only can't users modify or delete files under retention but administrators can not remove any file systems with FLR-C managed files present. There's a non spoofable clock in there as well to prevent people from trying to bluff the retention system.

So the question becomes when you offer FLR-C on Celerra why would someone use a Centera for compliance? For me economically the answer to that is pretty easy, when people have a compliance requirement for less than 25TB of file system data the TCO of a Celerra is quite compelling.

Above 25TB or if there are particular application integration requirements (Such as XAM support) then Centera not only has a better TCO but meets the application requirement where a Unified Storage wouldn't.

We'll cover something else in my next post.

EMC NetWorker Fast Start VSA

The NetWorker team have decided that a good way to show people what NetWorker (NW) looks like today is to make Fast Start available as a 1.0 Virtual Storage Appliance.

It's time limited to 30 days + 15 days of grace but that should be more than enough to get a feel for the system.

"So where do I get this modern wonder?" I hear you ask. This is where we have diverging paths.

If you have never used an Enterprise Backup product before I strongly advise that you click here. Then click the Evaluation Software tab, select what you want and you'll be taken care of by someone who'll get you moving.

If you're thinking to yourself "Hey Zilla, c'mon now it's me." then you can go right to the source and click here for the OVF files or here for the VMX files depending on if you are using Virtual Infrastructure Client or using Workstation/Server/Player.

Points of Interest.

A couple of notes for those going straight to the source. Inside those zip files are server and client images, both built on SLES 10 and running NetWorker 7.4 SP3.

The initial setup is all Q&A driven but if you're using DHCP addresses you're going to have to resort to the Sys Admin tool of champions, vi, to edit the /etc/hosts file and perhaps resolv.conf. (Though I knew I could skate away without it)

NW requires forward and reverse DNS to be configured correctly so if you're not running DNS you'll want to ssh into the systems login user=root password=welcome and make the appropriate edits.

Once that prep work is done you're ready to launch the GUI by pointing your web browser at the DNS name or the IP address with port 9000 and click on the start button to launch the java based NetWorker Management Console. (NMC) Login with user=administrator password=administrator 

This will start you in the Events window. You'll want to click the Enterprise button next to it. Select your server name from the left hand pane then right click on the system in the managed application pane selecting Launch Application. NMC can be used to configure and monitor countless NW servers so we've just selected the one we want.

When you launch the app it opens another window and you're now working with the server running in the VM. If you've configured your client and server correctly right clicking on FS_LINUX and selecting Start will begin the backup. Clicking on the Configuration button will allow you to fine tune the client configuration.

For recovery you have two options. From the client you can either choose to use the CLI based  recover utility and select files and directories from that or if you have X-Windows running and you have tunneling enabled with ssh just start nwrecover and use the Motif GUI.

I did all this through VMware player and it works fine, what doesn't appear to work so fine in the version of player as of writing is X-Windows running in the VMware Player window. I'm thinking a VMware tools install would handle this.

If you're using a Mac with VMware Fusion you probably have everything you need already installed. If not you can install all the Unix utilities from your MacOS X installation discs.

I did all this on Vista and having PuTTY and Xming makes life much easier.

Caveats.

-Thou shalt not use this for production backups.

-Though shalt not assume any warranty has been granted or implied.

-Thou shalt remember that you have accepted the license agreements. Like you could forget but don't think we don't know that trick about the EULA accepting Cat. Cat's can't drive backup servers. FACT.

-Thou shalt be clever and get this running with the Celerra Simulator as a Backup to NAS (B2NAS) target for a NetWorker Advanced File Type Device.

-And thou shalt always, always, remember that I'm not Chad and probably won't answer your questions on this. You never got those direct links from me and saying otherwise is just crazy talk.

Enjoy! :-)

Mr. Backup on TSM 6.1

Dubya-Cee-P (As he'll now be referred to forever) has a good write up on TSM 6.1 over at Mr. Backup.

I say it's good as I agree with it and in places is echoes what TSM customers have been telling me. If I disagreed with it I'd say it was bad.

Opinions are funny that way...

Emulex converges

HBAs remain an important component of any storage network so as a guy who helped install what probably was the first production FC SAN in Ireland I was interested in digging into Emulex's set of announcements this morning to see where they were in relation to the roadmap they had shown me when I visited them in the UK.

Pretty much on track from what I can see.

Their Gen 2 Universal Converged Network Adapter (UCNA) has been announced and that's probably the product to really put some wind into the FCoE sail. The Gen 1 product was a Gen 1 product. I don't have to explain that to anyone who's seen one up close but to anyone who hasn't it was a case of this all works, here's Ethernet, FC, iSCSI, etc but only in one form factor and we've yet to shrink all these chips into a smaller number of chips.

Gen 2 looks like it's been through the washing machine and is now the snug fit which will be popular with systems builders.

The timely announcement for me is the EmulexSecure Encryption HBA. This is an 8Gb FC HBA with on board hardware accelerated AES encryption. EMC offers PowerPath Encryption with RSA for data at rest use cases where you don't have hardware accelerated encryption. The common thread between those two offerings, one in hardware the other in software is the use of RSA Key Manager (RKM)

RKM is a very heterogeneous enforcement mechanism which spans multiple encryption technologies. It's also the key manager encryption technology vendors regularly look to for support.  We've seen a good example of that this week as if you've been reading the press you've seen that there have been three major encryption announcements and RKM has been part of all three of them.

RKM support for EmulexSecure Encryption has been announced. RKM integration is something Sun want for their open source KMS Agent Toolkit and RSA are going to deliver that. The next big encryption announcement this week was the official request to OASIS by RSA, HP, IBM and Thales to work towards delivering the Key Management Interoperability Protocol (KMIP) a common communication protocol for different encryption systems across various software applications and hardware devices.

RKM will support KMIP as RSA believes it'll accelerate the deployment of encryption technologies.

Sun has balked at KMIP as they feel they weren't consulted early enough. I think they'll eventually come along qw KMIP is a rising tide which will float all boats. For companies like Emulex KMIP simplifies their route to getting the Enterprise Key Management support their customers are going to need if they deploy EmulexSecure in a big way. For smaller vendors KMIP is their opportunity to add support from the big players a whole lot faster and with less fuss.

They are very different announcements but RSA Key Manager is at the heart of all of them.

And congratulations to Emulex for 30 years of excellence, I've only been using your products for the past 13 but no complaints from me. ;-)

XIV Drive Juggling

Storage Monkeys are asking about XIV useable capacity. That's a well walked path at this stage but what's interesting was a conversation I had with one of my NetWorker customers earlier this week.

I was onsite and we were going through NetWorker 7.5 and some of it's server virtualisation focused enhancements when over lunch we started talking about what was going on in the industry and he mentioned he had a Rev 1 XIV on the floor.

His observations from own testing were quite interesting. I've seen the XIV demo and they show off ease of use and allow customers to pull a few drives to show no data loss. Smart customers ones will pull two drives together and see if they end up with data loss, in the demo they don't but it turns out there's a trick to it.

Stay with me. We'll be going from obvious to complex to downright bizarre.

When you pull the first drive rebuild immediately starts for the data which no longer has two copies available. The less data in the system the faster the rebuild. This should be obvious.

Here's where it starts getting complex.

If you pull a second drive while the first is rebuilding the rebuild of the second doesn't start until the rebuild of the first has completed. If you attempt to access any data whose chunks are located on those two drives you'll get a fatal error.

If you pop the second drive back in it runs it's rebuild process and comes to the conclusion that it doesn't have to rebuild any data. The trick here is that the odds are in the XIV's favour that you won't be able to quickly locate the missing data so it looks like there's been no data loss.

And here's where it gets downright bizarre.

If I didn't know the person in question had an impeccable rep and a lot of time to torment any gear they're looking at buying I'd be looking for more than a pinch of salt with my lunch at this one.

He pulled two drives and replaced them, but he swapped their slots.

The box threw up dozens of checksum errors but then copied the contents of the newly inserted drive over what was already on the system. It replaced the correct data already in the system with the incorrect data from the newly inserted drive.

He said he was stunned when it happened and his team recreated it a couple of times to prove it wasn't a false positive.

Now, I don't know if this is a Rev 1 product thing or a XIV thing but if replacements are going to be handled by the customer we agreed that..well..this isn't a good thing. It's a bad thing.

A very bad thing.

(But at least he has a great backup product in place)

Flash: In a landscape

TSA does probably one of the best round ups I've read when it comes to the industry positioning of Flash drive technology.

And he did it on a weekend which highlights to me the kind of culture EMCers have. I've sent seven different people email since Friday night and have gone through dozens of replies since then.

The funniest one I read this morning from a guy in product development "writing this in restaurant bathroom. wife will murder me with a soup spoon if she finds out I took the crackberry to valentines dinner."

We love these technologies, we love this business.

HP caught out in a Flash

The latest evidence that HP still don't have a straight answer on the use of SSDs comes from this illuminating article up on InternetNews.com.

You can't just take that technology and use it as-is in a mission-critical datacenter application, Jimmy Daley, ISS marketing manager for HP (NYSE: HPQ), told InternetNews.com.

He's right you can't, but the work has been done and that's why I can buy 73GB SSD Array Groups for my XP24000 today if I call our HP sales rep.

I can pick up the phone right now and for HP's high end mission critical flagship storage system I can order a four pack of 73GB Flash drives and run production workloads on them.

A HP sales drone will not tackle me to the ground to prevent me from running part or all of a specific workload on an SSD Array Group nor will their support organisation hang up on me if I ring up and say one of the spares (Sold separately) has kicked in.

But it gets worse.

In general, true enterprise-ready solid state drives, we believe, will come later in 2009," with the products gaining widespread adoption by 2012.

So we're back to where we were before EMC announced the availability of the Enterprise Flash Drive in January 2008. Something we had been working on to make enterprise ready since 2006 and which caught a whole host of people flat footed as a result.

And customers are responding. While the Flash slow RAM Vs fast Disk debate goes nowhere customers are voting with their wallets and asking for larger drive capacities.

More Flash not less and today not tomorrow or 3 years from now.

Reading this is somewhat informative. It tells me that HP obviously think their SSD Array Groups are not enterprise ready and therefore should not be sold with the XP24000/XP20000 or used by anyone. Ever.

Or maybe it depends on who's speaking to the press at the time.