Storagezilla

Dive into Mozy Code

Mozy made some pieces of it's infrastructure code available in two open source projects last week. Now, Mozy operates at a different scale than many of us. They have more than 50PBs of data from more than 1 million customers stored in multiple data centers around the globe, as such I thought anything they could release would be niche but interesting to anyone considering operating at hyper scale.

It turns out after speaking with Reverend Ted Haeger over at Mozy I was only half right.

As per usual I go for the jugular with the first question.

99.999999% of the developers out there aren't operating at the scale you are and may not see a need for any this. Why release it?

We're big believers in Open Source Software and what we have over on Mozy Code are two projects that were started to solve problems specific to us but we think may now be useful in someone else projects. Even if it's only applicable to that 0.000001% of developers out there. They most certainly would have be useful for us to have had when we started out.

Lets go deeper into these projects starting with Ruby Protocol Buffers. I know that Ruby is an object oriented programming language but what's a Protocol Buffer?

A Protocol Buffer is a low fat data exchange mechanism over HTTP, invented by Google.

Lets go a bit deeper and consider the advantage of JavaScript Object Notation (JSON) over Extensible Markup Language (XML) object serialization: for each data field in an object, XML uses both an opening and closing enclosure (e.g. <fullname>Joe Tucci</fullname>). JSON is a bit more efficient, because it uses a comma to indicate the end of an object value (e.g. fullname:Joe Tucci,).

For a low use API, the smaller number of characters gives only a trivial advantage to JSON over XML. But as you scale up, XML begins to show its bulk relative to JSON, but JSON still carries redundant information: the "fullname:" label for the actual data could be eliminated. As long as the data gets transferred in a predictable position in a data stream, it doesn't need the label. 

That's what Google implemented with Protocol Buffers. They serialized the data in a more rigid, predictable format for faster communication between systems. As long as both systems understand the format, you can eliminate a bunch of overhead.

Google created Protocol Buffers to optimize data transmission in their data centers, where high speed data transfer between systems is all managed through lower-level languages. So the first implementations were in languages like C and C++. Naturally, Google also made a Python implementation, which was the first higher-level language to use Protocol Buffers.

Mozy is largely a Ruby shop for our web logic, but on the back-end, we're heavily C++ so at the scale we require Protocol Buffers made a lot of sense. However, we needed a working Ruby implementation to get that advantage. When our developers' tried a Ruby implementation they found that not only did it not work well for what we needed, but also that it would be easier for us to build it from the ground up.

And that's what we did.

So, the developer who would most likely be interested in our ruby-protobufs project would be a Ruby developer who needs rapid data exchange with back-end applications (likely written in a lower-level language) to reach a level of scale that cannot be so easily achieved using formats like XML and JSON.

XML and JSON will probably support the majority of developers' needs but ruby-protobufs is a straight forward implementation for anyone who chooses to use it.

Right, I can see where that would be useful to Ruby developers. Mordor is an I/O Library, how is it different from what's already out there? I've heard of threads, what's a Fiber?

Asynchronous I/O libraries have become well known tools of the trade, optimizing major limitations of synchronous I/O. 

Synchronous I/O requires programs to initiate a task (such as an HTTP GET, or file write operation) and then wait until the task completes. The application is blocked until the task completes.

Asynchronous I/O libraries allow a program to initiate a task, and then continue doing other things. When the task completes, a "callback" function typically kicks off a short routine for handling the task completion (such as doing something with an HTTP response, or recording success of the file write event).

Most current asynchronous I/O libraries use Threads as their base unit of execution. Threads allow simultaneous execution of tasks in a program, with each task using its own thread. This greatly increases program performance and resilience. For example, the Chrome browser uses a separate thread for each tab it runs, so a bad web app only hangs a single tab rather than the whole browser. And when you close a tab the memory it consumes is returned to the system.

But threads have downsides.

Because threads are managed by the OS kernel, the kernel allocates their memory and manages their scheduling. Mordor uses finer-grained execution units called Fibers. Conceptually, fibers are similar to threads, but instead of the kernel, the application schedules multiple fibers in the context of a single thread. This finer-grained approach to cooperative tasks reduces demand on the kernel for allocating, managing, and de-allocating. The result is much higher performance and lower demand on system resources.

The developers who would be interested in Mordor are those concerned with I/O task performance. Mordor provides developers with an Async I/O library through a simple API, with the performance benefits of a complex fiber-based system.

Great. All I need now is a compiler and I'm writing high performance code.

Well be sure to check out http://code.mozy.com before you do, you might find something useful!

Any more stuff coming?

These two are just the beginning. Going forward we have to consider that getting the right person to lead a project is as important as the technology itself. That's what will dictate what we release and when we release it. Right now there's great potential for future projects.

A moment with Luke Lonergan

Luke Lonergan is a co-founder of Greenplum and the CTO of the Data Computing Products Division at EMC.

The Data Computing Products division provides data computing solutions focused on "Big Data" processing.  These include the Greenplum Database, Chorus, Embedded Analytics, Hadoop and other kinds of processing infrastructure to be discussed later. Their latest solution is the recently announced Greenplum Data Centre Appliance. I quickly snapped a picture of two GP1000's on the manufacturing floor at the Cork Center Of Excellence as they were heading towards shipping.

 
 

It was interesting to see finished product on it's way out the door just a few days after I had spoken with Luke. I had some questions, he had some answers and as per usual I get to the point right up front

Is the Database market done? The Online Transaction Processing (OLTP) business has been long dominated by three commercial entities, isn’t that game now over?

Over? No. Technologically the “Database Wars” as we call them were fought by knights jousting in a field, by comparison the coming battle is going to be fought in Space. We're in the era of Big Data, hence the focus on really niche hardware designs from some as they struggle to square their existing technology with the needs of Online Analytics Processing. (OLAP) There will be different players and new winners in Big Data.

Define Big Data for me? It conjures images of an ocean of storage.

Well it can be an ocean of storage and some customers have an ocean of storage but Big Data isn't about quantity it's about gaining quality insight from multiple data sources. It’s about being presented with what you need to make a decision.

OLTP. OLAP. What's the difference?

Let’s take online shopping for example, OLTP is where you buy a product and it's recorded that something just happened. We sold something, decrement the inventory number for that item and get the order packaged and shipped. OLTP is the now. OLAP is the process of examining everything that's happened and gleaning insight from what's relevant. When shopping online you've probably seen suggestions for products they think you might be interested in as a result of your past buying behaviour…

I've seen varying degrees of accuracy and some very unusual suggestions with those things.

Right, some of those were organically developed from scratch over the course of a decade, we have people working for us who worked on some of the most well known implementations at the very beginning, but as you look at what followed you can see the accuracy getting better and better as the ideas were refined. These systems now suggest things I'm interested in looking at even if I choose not to purchase.

That's a consumer facing example, if we look at it from the business side let’s take the mobile phone provider. Customer churn is a massive problem as it costs a lot more to win a customer back after you've lost them than it does to keep them, but by examining a customer's usage behaviour you could offer them a more favourable deal before they dump you and sign on with a competitor.

In both cases quality information leads to a beneficial outcome.

OLTP & OLAP are two distinct categories right now do you see those merging?

Yes.

Is the Greenplum Data Centre Appliance (DCA) a step towards that merge?

Wow, is that the time?!?

Ha! It strikes me that the appliances in this space can be oddities in the IT infrastructure. They don’t integrate well with what people have already bought. 

That’s because of how they’re sold and I know this as I’ve been there. A Business Unit goes to the Operational IT team and says it has these particular requirements. Operational IT thinks about it, comes back with how long it’ll take to get approval, how long it’ll take to deploy, configure, test and put into production as well as what SLAs they can provide when it’s running. This process usually takes so long that the Business Unit is out the door and looking at other options. They’re not in the business of provisioning and managing infrastructure so they want this up and running in the next hour, not 6 to 12 months from now. They want the box with a power button on the front and a network cable coming out the back.

The strength of the appliance model is that it lands on the floor tested and configured at the point of manufacture, the weakness has been that many of these products are infrastructure islands. What’s good for the Business Unit can be bad for the Operational IT team as the appliance stands apart from all the existing production infrastructure.

What we’ve done with the Greenplum Data Centre Appliance is bridge that gap between the requirements of the Business Unit and the requirements of Operational IT. The DCA can be deployed and operated as a stand alone Appliance, turn it on and data goes in while decisions come out, but you can connect it to an EMC array if you choose, replicate it with RecoverPoint and back it up to Data Domain. You’re now storing the data on your production arrays, getting long distance continuous remote replication with bookmarking and backing it up to deduplication storage with built in integrity checking and bandwidth optimized replication. 

All of that means it’s no longer an island in your data centre. It’s part of the infrastructure.

It’s all industry standard components here. There’s Intel processors, disk drives and Ethernet. Nothing esoteric.

We don’t need anything esoteric. We put 16 segment servers in a rack and have qualified it to scale out to 24 racks. There’s 4608 Intel processing cores in that 24 rack configuration and we can use all of them. I'm not ruling out adopting anything beyond Processors, Ethernet and Disk Drives if it makes sense for us to do so but we get our performance from our Massively Parallel Processing architecture. What the appliance model does for us is increase our delivery options to customers. Stand alone software, appliances and virtual appliances for virtual infrastructure.

Are you working with VMware?

We have a number of active projects with VMware. In the near term we’re focusing with them on operation in purely virtualized infrastructure. That’s the starting point.

Tell me about Greenplum Chorus?

Chorus provides a collaboration framework for data analysis and sharing. We’ve wrapped it in an interface kind of like Facebook as we want to make it easy to create sandboxes and share documents and data between people of different skill sets who have different data requirements. We’re working to get the Chorus Beta ready for early 2011 so we’ll be talking more about that the closer we get to the Beta.

Greenplum and Cloudera made an announcement recently around MapReduce where’s the win there? I thought Greenplum already supported MapReduce?

MapReduce (MR) is like Assembly language for data programming. Greenplum has supported a MapReduce interface directly into our parallel execution engine for several years. Our relationship with Cloudera allows us to support even more configurations of Greenplum and Hadoop, Hadoop being one of the popular flavours of MR, but programmers can need a higher level of abstraction so our objective is to facilitate alternative bindings for Perl, Python, C, Java and other languages in a naturally parallel manner.  Our high level SQL interface is more powerful for the experienced data professional, and in addition to SQL and MR we are providing other powerful programming abstractions.

Got any Scott McNealy stories you want to share?

Well that depends on how much Beer you have...?

NFSv4 Vs NFSv3? FIGHT!

Stephen Foskett asks the question in “Is NFSv3 really that bad?” but then goes off on a tangent and compares it to pNFS.

Let us be clear, pNFS is only an optional feature of NFSv4(.1) NFSv3 should be compared directly to NFSv4.

Off the top of my head here’s what NFSv4 brings to the table when put head to head against NFSv3. This is not an exhaustive list and I’m doing it from memory so you’ll excuse anything I’ve left out.

	NFSv3	NFSv4
Exports	All exports are mounted separately.	All exports can be mounted together in a directory tree structure as part of a pseudo-filesystem.
Protocol	Numerous protocols for different aspects collected together. MOUNT, LOCK, STATUS…etc.	A single protocol with the addition of OPEN and CLOSE for security auditing.
Locking	Permanent locks in yet another protocol.	Lease based locking in the same protocol.
Security	UNIX based. SecureNFS. Mode Bit Locking.	Kerberos and ACL based.
Communication	One operation per RPC.	Multiple operations per RPC. (Improves performance)
I18N	All locales must match.	UTF-8.
Parallel high bandwidth access	None native. (Addition such as MPFS)	pNFS.

So looking at the list above we see some dramatic improvements, modernizations and simplifications in NFSv4 over NFSv3 as well as the addition of optional parallel high bandwidth access.

But we also see things like deep security functionality which people may not require as well as the fact NFSv3 is stateless and will therefore work better on crufty networks while NFSv4 isn’t and won’t. Also, while NFSv3 is a bunch of protocols working together it’s a bunch of simple protocols working together and it’s found everywhere as it’s trivial to implement. NFSv3 has a long life ahead of it yet but the killer feature for NFSv4 was delivered in v4.1. pNFS.

pNFS will drive NFSv4 adoption as it is transparent to applications and designed for High Performance Computing use cases.

Do we need pNFS?

I don’t think anyone doing Seismic Data Processing, Video Streaming or anything like that requiring access to PBs of storage at GB/s speeds ever said they had enough throughput. I believe pNFS will be first adopted by those segments and it’ll drag NFSv4 with it.

1+1 = 3 Even Before OpEx Discount

Increasingly, the new field of battle is in deeper integration between backup software and dedupe storage. Avamar started this wave a long time ago, and it remains the most advanced implementation. More traditional backup software is also moving in this direction:

• Symantec pioneered the idea using OST and Data Domain Boost to integrate NetBackup, and now Backup Exec 2010 with Data Domain and other OST licensees.
• Starting this summer, NetApp and SyncSort are offering channel-based integration packages for SyncSort to manage snapshots on FAS systems. Not quite Boost, but reacting to the same problem.
• Also starting this summer, Symantec are offering their own storage systems in some geographies for PureDisk. (I love OST, but how weird is it to be on the NBU 5000 team in the “No Hardware Agenda” company? It would be like working in DB2 support at Oracle.)

via www.dedupematters.com

More over at Brian's blog.