Data reduction redux: What's in a name?

May 7, 2009 – As mentioned in a previous post (see “Welcome to the data dedupalooza” ), I’ve been researching the topic of data reduction for primary storage for an upcoming article. One thing I’ve noticed is that the industry is playing fast and loose with the term ‘data de-duplication,’ sometimes using it in place of the more general ‘data reduction’ or ‘capacity optimization’ terms.

Maybe that was ok in the realm of secondary storage, but as you begin considering data reduction for primary storage you’ll want to be more specific about what technology a vendor uses, although all of the technologies produce the same result (to varying degrees): reduced storage capacity and costs.

In the secondary storage space, in addition to data de-duplication, some vendors use single instancing or compression or a combination of techniques. In the primary storage space, some vendors use data de-duplication, some use compression only, and some combine the two.

Again, all of these technologies have the same goal, but in the case of primary storage (and depending on the type of data sets you have), it may make a difference how a vendor is implementing data reduction. And it’s misleading to refer to any type of data reduction as ‘data de-duplication,’ despite the popularity of the term.

If you’re interested in data reduction (aka capacity optimization) to reduce your capacity requirements, delay new purchases, and slash power, cooling and space requirements, plan to attend our upcoming Webcast, titled “Leveraging Capacity Optimization to Reduce Datacenter Footprint and Storage Costs," Tuesday, May 12 at 10:00 PST, 1:00 EST. Presenters will include Noemi Greyzdorf, research manager, storage software, at IDC, and Peter Smails, senior vice president of worldwide marketing at Storwize.

The Webcast will cover data reduction for all types of storage -- including primary, nearline and secondary data – and provide tips on how to maximize the benefits of capacity optimization. To register, click here.

Welcome to the data dedupalooza

May 1, 2009 – I’ve been looking into data reduction (or capacity optimization) technologies for primary data storage recently, including data de-duplication, compression and others. Of course, compression and de-duplication are well-proven and now common on secondary storage devices such as VTLs and other backup/archiving systems.

However, despite the fact that vendors such as NetApp and Storwize have been selling (or, in the case of NetApp, giving it away for free) data reduction software tuned for primary storage for some time, end-user adoption has been relatively slow.

It looks like that’s about to change.

For one, EMC’s entry into the market with (free) data de-duplication and compression for its Celerra file servers lends legitimacy to data reduction on primary storage devices. See “EMC adds de-dupe, SSDs to NAS.”

Two, NetApp is seeing some real traction in this space. The company claims to have shipped more than 30,000 systems with its ‘NetApp deduplication’ technology. The question is how many of those customers have actually fired up the data de-duplication functionality. The answer is “at least 15,000,” according to Larry Freeman ( aka Dr. Dedupe ), senior marketing manager for storage efficiency at NetApp. And it’s probably a lot more, because the 15,000 are only the ones that NetApp can track via its autosupport feature, which not every customer uses.

Since NetApp can de-dupe across all of its storage platforms (including FAS, V-Series, and VTLs), the next question is: Of those 15,000+ systems that are actually running de-duplication, what percentage are primary systems? That one’s difficult to track, but Freeman estimates that “about 60% of those systems are de-duping at least one primary application.”

Interesting stats, but any way you slice it the entry of EMC, and the success of NetApp, in data reduction for primary storage means that this market is poised for takeoff – despite end users’ lingering concerns about performance and data availability and reliability issues which have, for the most part, been addressed by the vendors.

And a rising tide lifts all boats (maybe), which is good news for the smaller players and startups in this space, including vendors such as greenBytes, Hifn, Storwize and Ocarina Networks (which recently inked about six partnership agreements).

All of the vendors go about data reduction in very different ways, but the end goal is the same: A reduction in primary storage capacity, which results in cost savings and/or purchase deferments, as well as power, cooling and space savings. In this economic climate, who can argue with that?

More on this topic as I delve deeper, but for now . . .

The best article that InfoStor has posted on this subject is “Primary storage optimization moves forward,” by the Taneja Group’s Eric Burgener.

Also check out SNIA’s Data Management Forum’s Data De-duplication and Space Reduction Special Interest Group.

And sign up to attend an upcoming InfoStor Webcast, titled "Leveraging Capacity Optimization to Reduce Datacenter Footprint and Storage Costs," Tuesday, May 12 at 10:00 PST, 1:00 EST. Presenters will include Noemi Greyzdorf, research manager, storage software, at IDC, and Peter Smails, senior vice president of worldwide marketing at Storwize. The Webcast will cover data reduction for all types of data, including primary, nearline and secondary data. To register, click here.

Consider data de-duplication plus compression

Data de-duplication gets a lot of ink and engenders a lot of debates, but the option of combining data de-dupe with real-time compression on primary storage is rarely discussed, maybe because there aren’t many vendors targeting compression for primary storage resources.

That’s what Storwize specializes in, and the company recently released the results of tests that suggest a serious look at combining de-duplication with data compression to further enhance overall data reduction.

Storwize claims that adding real-time compression can improve data reduction by more than 200%, based on its internal testing as well as customer deployments. (In the internal tests, Storwize used data de-duplication solutions from both Data Domain and NetApp.)

Storwize positions data de-duplication as being particularly advantageous for highly redundant backup data sets, but points out that using real-time data compression on the front end (primary storage devices) can significantly reduce de-duplication processing time and enhance throughput, in addition to reducing overall capacity requirements.

To assess the validity of the test results, check them out here

And if you’re among the early adopters of data compression on primary storage, respond below or email me at daves@pennwell.com and let me know what types of real-life data compression ratios you’re getting.

Dr. Dedupe blog

In previous posts I've noted that data de-duplication will be one of the, if not the, hottest storage technologies in 2009. For a humorous (and, well, odd) look at data de-duplication, check out NetApp's Dr. Dedupe Blog.

Better yet, check it out on YouTube.

Top storage technologies for 2009

It’s time to ring in the New Year with the obligatory Top Storage Technologies for 2009. I’ve culled through dozens of picks from the vendor community, and the one that comes closest to my own predictions is from HP. So, according to me and HP, here are the hottest storage trends for this year, in no particular order:

Cloud-based storage is changing how companies commoditize and store the exponential amounts of data from video sharing, digital media and social networking sites. The need to store massive amounts of data with real-time access capabilities will change the storage industry in 2009 as we see a greater uptake in cloud-based storage solutions that address the challenges of data growth, but at a fraction of the cost of traditional storage infrastructures.

Data de-duplication. By drastically increasing disk utilization (some vendors claim data de-dupe ratios of 50:1 or more, although your mileage will vary), data de-duplication reduces storage costs and improves capabilities such as disaster recovery.

Solid-state disk (SSD) drives: In a recent InfoStor QuickVote reader poll, only 6% of the respondents were currently using SSDs; another 6% planned to purchase SSDs within the next six months; 14% within the next six to 12 months; and 74% had no plans to use SSDs. However, according to HP: “Once the industry addresses the cost premium and capacity challenges of SSDs, we will be one step closer to IDC’s predicted 70% compound growth rate for SSDs by 2012.”

Virtualized storage: IT organizations are looking to virtualize their storage environments to achieve the same benefits they’ve seen from server virtualization. This is leading to “shared data” architectures that ease storage management burdens.

Secure storage: Industry and government compliance requirements, as well as the need to protect sensitive data, will put storage-level security at the forefront in 2009. More and more users will opt for encryption, including drive-level encryption, to protect data.

Energy-efficient storage: Technologies such as data de-duplication and SSDs eliminate the need for unnecessary hardware to improve performance while reducing power consumption.

2008: The Year of . . .

It’s customary to take a year-end look back at the key trends and technologies that shaped, or re-shaped, our industry. In storage, it’s usually difficult to zero down on one or two topics, but in the case of 2008 it was relatively easy.

In my view, two technologies dominated this year: virtualization and data de-duplication.

In the case of virtualization, I’m referring to server virtualization and its impact on storage, rather than storage virtualization per se (although the two are careening toward convergence). Server virtualization is arguably the hottest trend in the overall IT market, and virtually all storage vendors are hustling to take advantage of this trend by providing integration hooks and even brand new products that help end users maximize the benefits of server virtualization.

With Microsoft’s introduction of Hyper-V this year, which will ignite competition with VMware and other platforms, adoption of virtualization will continue to surge, and the storage implications – from HBAs to backup software – will take center stage for storage administrators in 2009.

But I’d have to give top honors for the dominant storage technology of 2008 to data de-duplication.

The rapid rise of data de-duplication is no surprise: From the end-user perspective, it’s a no-brainer. There aren’t any cogent arguments against reducing your capacity requirements and costs, particularly in these belt-tightening times.

But the thing I really like about data de-duplication is that it’s one of those technologies that show how end users win in the end. Data de-dupe works against the profit motive of vendors (at least the hardware vendors) because, like thin provisioning, it enables users to significantly reduce the amount of storage space they need to buy – or at least defer purchases. And let’s give credit to the early pioneers of data de-duplication, such as Data Domain, that gave users a technology that was so useful that they in turn forced the big vendors to offer a technology that in a way was not in their near-term best interests.

Here are some recent InfoStor articles on data de-duplication:

Users weigh de-dupe options

Users react to de-duplication deals

EMC refreshes data protection, de-dupe software

The hottest technologies at SNW

In my last blog I promised to reveal The Hottest Technology at last week's Storage Networking World (SNW) show but first, the runners up.

5. Solid-state disk (SSD) drives. There was a lot of talk about SSDs at SNW (mostly in the context of the raging SSD-vs.-HDD debate), but there weren't many product introductions at the show, although Intel did announce production shipments of its new line of enterprise-class flash drives, which will eventually spur further price erosion in this yet-to-get-hot market.

4. Cloud-based storage. Again, a lot of talk, but few vendors. The only cloud storage vendor I met with was Nirvanix, although this services category is expected to grow rapidly over the next couple quarters. For more on this subject, see "What is cloud-based storage?"

3. Storage efficiency technologies, most notably data de-duplication and thin provisioning. Data de-duplication for secondary storage is becoming widespread, but there was a lot of talk at the show about data de-dupe for primary storage from vendors such as NetApp, Storwize, Ocarina Networks, and there will be much more to come.

2. Server virtualization. This is clearly the dominant IT trend, but there are so many storage technologies focused on optimizing virtual servers that it didn't quite make the top of my list.

And the winner is . . .

Fibre Channel over Ethernet (FCoE). This is admittedly an odd choice because, in terms of end-user adoption, FCoE may be years away, but

For one, FCoE was the topic of the only major press conference at the show (hosted by QLogic, Cisco, NetApp and VMware).

Two, a number of vendors made FCoE product -- or at least product certification -- announcements at the show (albeit prototypes in most cases), including EMC, NetApp, Brocade, Emulex and QLogic.

And three, FCoE will provide controversy (as in 10Gbps iSCSI over Ethernet vs. Fibre Channel over Ethernet) for years to come.

I'll address the controversial side of FCoE in my next blog.