Storagezilla

How to rig the game when asking for new software features.

One of the biggest time wasters I’ve come across in my working life are meetings. That’s not to say meetings aren’t important, they are, but when badly run they’re torture.

Meetings with your co-workers or partners or customers are very important but meetings shouldn’t be about deciding things, the decision making process at the top is much less public than anything you’ll ever see presented in a meeting format, so meetings should be about communicating what’s already been decided and taking feedback on that and/or assigning tasks to carry out the decisions that have been made.

Since I moved out of technical pre-sales and into a more strategic thinking role a lot of my time and nearly all of my travel time is spent communicating to co-workers, partners and customers what has been decided and how we are going to proceed.

I may have had some direct input into a few of those decisions, for a lot of others I probably won’t have, but I always attempt to dig out the ‘why this and not that’ of any of the major decisions and as a result I end up (usually) with a well run meeting. Well run doesn’t mean neat & tidy, good meetings are messy, what it means is at the end of it people are clear about where things are going and why we’ve decided to go there.

In such meetings feedback is always welcome but it needs to be the correct kind of feedback.

Lets say there’s an issue of some sort. A piece of functionality that needs to be added or changed to act in a different way. If you can cleanly define the issue and do it in an amount of time that doesn’t put people who don’t care too much about this peccadillo into a coma you’ve started well.

That’s the easy part and being direct it’s the incredibly lazy part.

Where you need to start really thinking and what you’d need to be able to communicate clearly, pictograms and cave paintings are acceptable, are things such as: 

-What you think the solution should be?

-What alternatives have you considered?

-Is your solution reactive because some other product out there does this right now? If so are you solving a real problem or just checking a box?

-How flexible are you to the solution being totally, completely and entirely different than what you’ve proposed if it solves the problem?

That last one is actually a major challenge with some people. If you can crack the problem but don’t do it in a way they think it should be done, they come back to you with your solution presented as a new problem. It isn’t, it’s a solution to the problem they highlighted, probably implemented in a technical way that makes sense for everything else you’re trying to achieve in the context of your product vision.

You’re never going to get everything you want into a product release and have it ship with high quality and on time. The list of things you can do is dwarfed by the things you want to do. Both of those being tiny in the face of what you realistically can do.

That’s why it’s important to have thought out from end to end what you’d like done before you ask for it.

The easier it is to say yes to a coherent well thought out piece of feedback the greater the odds things will get done.

Stuff your predictions.

The truth may be out there, but lies are inside your head.

--Terry Pratchett

Yes my friends, it’s that time of year where all the self serving lists of tech predictions appear.

The question I have is ‘Why now?’. Why now when if you look at the track record of the folks pushing their 2013 prediction lists throughout the year, they can barely get what’s happening this month correct.

Just because it’s Hogswatch doesn’t mean they’ve now been granted precognition powers along with their delivery of Hogswatchnight meat.

To save wear and tear on your eyeballs, better spent reading something (anything) else rather than vapid predication list nonsense, I’ll break the majority of these lists written by anyone in technology who’s selling something and that includes themselves, down for you. That way you can get on with the important business of eating chocolate and drinking booze for breakfast while telling yourself it’s okay because ‘It’s Hogswatch.’

Here we go:

• Whatever that other guy is trying to sell to you is awful. You’ll be sorry to have bought it. It’ll probably give you a sexually transmitted infection, your teeth will fall out and your partner/spouse/children will see you for the clapped out waste of skin you’ve always been.
• Whatever the predication list writer is selling is amazing. This point will be proven, proven mind you, by the marketing hoopla they’ll create in the coming months. Because if they’re talking about it then obviously everyone is talking about it.
• A point incredibly obvious to anyone who’s capable of independent thought but now tarted up and put out on a street corner as something unique and insightful..goes here.

And there you go, that’s your 2013 predication template right there just pad it out with as much filler as you can think of.

I’d rather prediction list writers use their amazing new powers to provide me with the lottery numbers instead.

But that would be a Hogswatch miracle.

Cheap heat, no mojo.

Looking to generate some cheap heat (And because EMC doesn’t speak to him at all) Robin Harris has started making things up about Project X.

The core of his argument appears to be that if it wasn’t going to be incredibly late by his estimate, the company would have conducted private betas by now. But since he hasn’t heard of any they must not be happening.

Well that’s a valid point, so valid that EMC actually allowed the results of a customer POC  to be posted online by the team who performed it for the customer back in September.

September.

This is a case where you can’t do someone’s reading for them.

He could also have read the hands on Storagereview.com did with the product at VMworld 2012 back in August. You’ll notice the final line clearly spells out availability in the first half of 2013 just like Chuck said in December.

I think by the end of his post we get to the crux of the matter, there’s bitterness that EMC didn’t buy one of the companies he has a relationship with instead.

All involved should be aware that if their strategy is to hope it takes another year to get this out the door they’re going to be terribly, terribly disappointed.

Lotus: Killed by death.

‘Make him feel that he is dying.’

-Caligula

Putting aside Gaius Caligula’s sadism in his instructions to the executioners of his victims, this week there are more than a few companies feeling like they’re dying. Without me personally condoning or condemning I’ll let you decide who those are.

You have the autonomy to do that.

What did catch my eye this week was the wave of nostalgia which broke upon the shore at the news of IBM retiring the Lotus brand. I find nostalgia in technology to be pointless at best and dangerous at worst. You work in an industry which can only move forward even if you think you’ve seen an idea before, that doesn’t mean you forget the past but you should make sure to keep it there.

When I think of Lotus I think of two things.

1- It’s been driving growth of Microsoft Exchange for over a decade and now that’s over. Exchange is taking numerous shots to the groin from Google Apps and the like while the Fortune 500 is pretty much tapped out for people who might be interested in migrations to Exchange. We’ll see what happens when there are no easy Domino server and Notes application swap outs left.

The brand might be gone but the products are not. Though if you’ve ever suffered under having to use Notes you most certainly wish they were.

2- Bill Gates had waltzed Lotus down the acquisition aisle back in the 80’s only to find his bride to be had cold feet and she took off as fast as she could in the opposite direction. Gates-era Microsoft (Back when Microsoft was a terrifying competitor, not the case today) did what Gates-era Microsoft always did. It wrote a lot of code, bought some code, (PowerPoint was a 1987 acquisition) created MS:Office demolished Lotus SmartSuite and then circled back with Exchange to conquer the Groupware market Lotus created.

Regardless of if it was IBM or Microsoft, Lotus had it’s card marked as an independent software provider years ago. That’s not to say they didn’t have some of the brightest application developers in the world.

Some of the coolest people I knew worked for Lotus but while whip smart and with it they had the sense of urgency of a glacier in the middle of an ice age. That appeared to be the de facto company failing.

All in all don’t mourn for Lotus.

It was too hip and too dysfunctional a company to live.

Software development is art. Not science.

Storagebod had an interesting post asking why software sucks and why we don’t just slow things down and wait until it’s done right.

I’ve thoughts of my own not directly related to his post but of course I want to share them.

Starting at the end, software is never ‘done’. It’s never completed. Hardware can be completed because hardware development is an exact science rooted in physics and the results are therefore repeatable when the science is applied correctly. Software development is not a science, it is an artistic skill in the way that writing a novel or painting a picture is an artistic skill.

The difference between a hardware engineer and a software developer is the difference between Johannes Gutenberg & Ernest Hemingway.

Both important to the typeset word in general but much different in specific.

Keeping in mind with software never being done the best way to improve any piece of commercial software is to ship it and then iterate on it. I don’t know of a software developer who wants to rush anything out the door, if anything they’d never ship anything they didn’t think was a perfectly formed snowflake if they could help it, but following that to it’s logical conclusion we end up with a much poorer choice of products in the market and a lot of never seen code held hostage by it’s creators who are in thrall of the masterpiece syndrome.

When it’s shipping, it’s a real product. A product you hope is used by people and therefore worthwhile. If it is used by people you’re motivated to carry on and make progress with it because the realities in commercial software development are if you’re not adding features you’re losing paying customers.

The folks spending their money to put shoes on your kids feet, or a roof over your head, will just go spend it with someone else.

I’m not quoting from Martin or inferring anything on his part but I’ve typically found there’s an aspect of wanting your cake and eating it too to these things. If you just focus on your existing code you’re told your product is ‘uncompetitive’ in relation to others.

They have feature X. When are you going to have feature X? The 2013 version of Y came out an hour ago. We’re thinking of upgrading. Why don’t you support the 2013 version yet?  When are you going to support it even though by asking about it I’m not saying we won’t decide later that upgrading is a waste of our time and instead we’ll sit on the previous version for so long we’ll want you to extend your support of it?

While if you do continue to add features and work on product quality you’re usually still held to standards well above and beyond the development and financial capabilities of your resources. So all you can really do is do as much as you can.

The defence and space exploration industry is a prime example of the slow it all down to ensure it’s right the first time development mentality. That’s why the Curiosity Rover is making it’s way across the Martian landscape with a brain running at a leisurely 200 MHz and it required 4 days to update the 27 year old system software just to enable it to drive down the landing ramp.

It should be noted that the 4 days wasn’t transmission time in the vein of an intergalactic patch Tuesday download from Earth to Mars, the four days was from when they started the installer on the code when it was fully downloaded to the Rover.

The people involved need to get everything correct the first time as sending someone over to give something a quick reboot just isn’t going to happen. The result of that requirement of course is that programmer productivity goes way way down. There’s a lot of programmers writing very little code but doing  a hell of a lot of testing on the code they do write. That’s acceptable because it suits the needs of their operational environment but they have a specialised set of needs (Average journey 225 million Km. No breathable atmosphere. It gets wicked cold at night.) uncommon in IT.

So, my prime example of the slow software movement exists today. Everything from the phone in your pocket to the TV in your living room is an order of magnitude more complex. Now imagine your most intensive production application and think of how much more complex that is.

Is slowing it all down the answer? Well, if you want to continue to use systems that were state of the art ten years ago it could be your personal answer…

The Future Of Backup, with Simon Seagrave.

I spoke with Simon Seagrave about the future of backup over on his blog. I like Simon, he’s a very good interviewer and technically understands the topics he’s asking about.

That’s more unusual when it comes to people with video cameras then you’d think.

LTFS: Reflections in a Zilla’s eye.

Every now and then, but not as frequently as you’d imagine, I get asked about LTFS support so I was interested to read Chris Evans comments from IP Expo and have one or two quick comments of my own in response.

The questions I get tend to be in the context that ‘my vendor says this is the greatest tasting soup ever’ and a quick drill down leads me to suspect people never get around to the implementation today part of the conversation, but that’s a different post.

To declare the personal bias upfront I absolutely wake up in the morning and believe that the LTFS tape as disk concept is ridiculous and borne out of desperation. I won’t elaborate on that as that too is a different post but having declared that personal bias I’m going to walk away from it for now and take a more analytical view. 

Looking at the reflection in a Zilla’s eye when we size up vendor support for LTFS what strikes me is we have now three drive manufacturers writing their own driver software for each platform they want to offer the functionality on. So the interface software burden moves from the backup software provider to the backup hardware provider. Maybe if you're a backup software provider you think that's a good thing, or not, it's a personal opinion as to if that's where the coding should be done, what the update schedule should be and how long you’re willing to wait for support to be current?

Moving from where the interface work should be done and looking at formats, why do backup software vendors have proprietary formats? Quite simply because to their information management system (Backup software) they need to represent hundreds of thousands, millions, tens of millions, billions, tens of billions (and so on) of files in some form of file system (I'm being very loose with that term) which is distributed across various different pieces of media, some of which could be offline, where the native file systems of each could not support anywhere close to that file density used directly.

Right now with each LTFS cartridge being it’s own piece of DAS, like it were an external hard drive or optical disc, managing those won’t be too bad when you have a handful of them. When it’s a box or a crate of them you’re going to want another layer of software over the top. And before you know it, we’re back to backup applications and their need to represent everything as a consistent file system all the time regardless of the active/offline state of physical components of that file system.

There are very solid technical reasons as to why backup applications developed in the fashion that they have and many of those technical reasons remain unchanged to this day.

The incredibly glib but correct answer to solving the challenges involved in differing formats and data mobility is to actually backup less data and retain it for much shorter amounts of time.

When we hold onto information for more than a year we might as well be holding onto it forever.

Slow death through fast feedback

Not being someone who should be left behind the wheel of a fine automobile ever (I’ve been pulled over on nearly every civilised continent on earth for ‘unusual’ driving), there’s an interesting comment from the former CEO of Porsche, Wendelin Wiedeking.

What it boils down to was is if he listened to exactly what Porsche customers said they wanted and he built to that feedback, he’d be selling a badly knocked off BMW and that isn’t who his company is. So he felt he was better listening but only doing the things which fit to a vision the company developed without consultation.

It’s not that customers are wrong, they’re all right in some way as they’re looking to address an issue which affects them, be that a problem of extra leg room or getting a handle on a heterogeneous computing environment, but the problem with following customer feedback directly is that there’s a slow incremental creep towards mediocrity until eventually you end up with a bland, grey, gelatinous uninspiring product which could be a poor echo of someone else's stuff.

I talk to a lot of people every day of ever week I’m working. (I need to take some holiday time, but the concept of going to the airport to fly somewhere strikes me as being too much like work) People who develop, market, sell, support, use and live with products but I’m finding more and more I have to keep Wiedeking’s comments in mind. If we can’t come up with things ourselves that people didn’t know they needed before we showed it to them, we’re just selling the same uninspiring product as anybody else.

Fusion Drive, auto tiering goes mass market.

Apple’s Fusion Drive very much appears to be auto tiering for the mass market, (You’ll remember auto tiering from 2010 when NetApp said it was dead) but what worries me is a lack of RAID protection and the unknown right now around what happens if the Flash Drive physical component or the Hard Drive physical component of a logical Fusion Drive, fails?

We can’t say you lose everything as this could be more a file system namespace where each storage component is accessible independent of the other (Like individual storage containers in DDOS using Extended Retention) and less a concatenated disk set. Those details will out very quickly.

As you might know when Flash Drives fail they tend to just blink out like a blown light bulb, assuming Apple are using the endurance technology acquired from their Anobit acquisition they’ll certainly claim the other parts of the system could go before their Flash drive does. But again, like a light bulb. It’s not like hard drives where funny noises and other failing electro-mechanical randomness can give you a heads up that service is about to be withdrawn any day now.

In general if you’ve thrown all your data into a single SSD bucket you need to be on top of backups. Bad things happen to good people for no other reason than that they can, but as we come out of the early adopter phase on the consumer side of things we’re going to be shaking out the quality from the crap real soon now as the technology in use ages and gets some miles under it’s tires.

Take those backups of your personal data to a hard drive, optical disc, punch card, tape, the cloud, whatever you have to hand and works for you so you can make sure it matters not who your device manufacturer placed their bet with early on.

Quick side note, in this case keeping HFS+ kicking around actually allows Apple to get out in front. Microsoft’s NTFS replacement ReFS and the underlying Windows Storage Spaces with it’s Slab extents is an attempt to do a storage controller for the DAS Microsoft want you to run Exchange & SQL Server on. As such auto tiering on Windows with ReFS and Storage Spaces could take a while longer as the concept is more ambitious and also aimed at the Enterprise, not the desktop.

But it will happen.

The road to Damascus

You go out and talk to customers and partners and you come back with a lot to chew on.

I’m always in two minds about that feedback, on one hand I absolutely want to give everyone involved what they need to make them successful. On the other I wonder if that feedback prevents me from going off and pushing hard for the stuff they don’t know they need yet.

It’s a delicate balance.

One of the really positive things from this round of trips is I no longer have to take shelter behind the lectern anytime the conversation turns to NetWorker. Back when it was shamefully neglected, under funded and largely ignored by the powers of the day you’d rightfully get blasted for showing up with a rather threadbare list of improvements and catch up features.

This was at a time when you could clearly show that Avamar and Data Domain were charging ahead with net new features and moving into new spaces.

At the time I used to be as exasperated as the user base, but you were damned with faint praise anytime it looked like you were moving in the right direction.

It’s a massive change to move from the ‘You guys suck with that product’ (I know) comments of two years ago to the ‘What you’re saying you’re going to do sounds good, but I’ll want to verify it myself’ (Of course) of last year and arrive at what I’ve heard from people over the past few days.

‘You’re now on the right track. Stay on it. Do more of this.’ (That’s the plan)

In one case just by upgrading from the old 7.x release of code to 8.0 and without making any other changes one customer saw a 20% increase in overall throughput in his environment. A 1/5th gain in overall performance just by running the 8.0 installer. That was before he dug into any of the new features.

As a rule I don’t make promises to customers or end users (or anyone at EMC) because I know that like the biblical road to Damascus the software development process is long, winding and can lead to conversion along the way, but what I’ve been committing to people as I know it to be true, is that all of NetWorker will be renewed and refreshed in the coming 8.x releases.

Root & branch.