Musings on the future of data dedupe

December 10, 2010 – I recently chatted with a few vendors in the data deduplication space. As conversations often do at this time of year, the talk turned toward the future of data deduplication. Here are a few snippets.

“Tier 1 storage vendors will move past point solutions for deduplication next year,” says Tom Cook, CEO at Permabit. “They’re working toward end-to-end deduplication, across SAN, NAS, unified [block and file], nearline and backup.”


“When that happens, once their customers ingest data and get it into a deduplicated state they’ll never have to re-hydrate that data throughout its lifecycle. The data will stay deduplicated through processes such as replication and backup. That’s a huge savings in workflow, footprint and bandwidth,” say Cook.

“Today, the big vendors use a variety of point solutions, but they’d like to use a single data optimization product across all their platforms, whether it’s block or file, primary or secondary. End-to-end deduplication will creep into the market in 2011 and 2012,” Cook predicts. (Permabit sells deduplication software – dubbed Albireo – to OEMs.)

Personally, I don’t think that single-solution, end-to-end deduplication will happen that quickly, in part because of the huge investments that the Tier 1 vendors have made in their “point solutions,” but we’ll see.

Dennis Rolland, director of advanced technology at Sepaton, has some predictions that are similar to Cook’s, as well as some differing opinions regarding trends in the data deduplication market.

“Dedupe will be required in more places going forward, including primary storage in addition to nearline storage, and end users will have to cut down on how many dedupe solutions they have because of the complexity in managing many disparate solutions,” says Rolland, “but we’ll probably still have distinct solutions for primary and nearline storage deduplication.”

Rolland thinks that the emphasis on deduplication benefits such as capacity, footprint and cost savings is shifting. “Dedupe enables low-bandwidth replication, which in turn enables companies to economically deploy DR [disaster recovery] sites,” he says.

Rolland also links two technologies that will no doubt make my list of The Hottest Storage Technologies for 2011 (assuming I get around to making such a list): data deduplication and cloud storage.

“Dedupe is an enabler for cloud storage,” says Rolland. “It makes it practical to deploy cloud storage because you’re sending, say, 10x less data over the WAN. That has significant implications for deploying cloud-based DR.”

(Sepaton bundles data deduplication software with its virtual tape libraries, or VTLs.)

Meanwhile, Quantum released the results of an end-user survey this week that suggests U.S. companies could save $6 billion annually in file restore costs by adopting deduplication.

According to the survey of 300 IT professionals, respondents spend an average of 131 hours annually on file restore activities, with 65% restoring files at least once a week. Based on the average wage for IT professionals in the US ($31.55 per hour according to PayScale.com), that equates to $9.5 billion. However, Quantum’s survey also found that those companies that are most efficient at file restoration predominantly use deduplication and can complete restores in approximately one-third the average time of all respondents. So, according to Quantum’s press release: “If the broader US market was to achieve similar data restore efficiencies, the potential annual savings for US businesses would be approximately $6 billion.”

This survey seems a bit misleading to me because it’s not really focused on the advantages of data deduplication per se in a file restore context but, rather, the advantages of disk-based backup/recovery.

Steve Whitner, Quantum’s product marketing manager for DXi, explains: “If you back up to regular [non-deduplicated] disk and you have a need for DR, you have to get that data to another site and you can’t keep data on conventional disk for very long – maybe a few days or a week. So the real issue is not the speed of restore; it’s the fact that companies can now store a month or two of deduplicated backup data on disk.”

You be the judge. Here’s Quantum’s press release and here are some supporting slides from the survey results.

One thing is clear: In 2011, the focus will shift from deduplication for nearline/secondary storage to deduplication for primary storage. Witness two of this year’s biggest storage acquisitions: Dell buying Ocarina Networks and IBM acquiring Storwize. (Storwize’s technology is now in the IBM Real-time Compression business unit.)

Related blog posts:

What is progressive deduplication?

Data deduplication: Permabit finds success with OEM model

Who will be acquired next? And the Top 10 are . . .

UPDATED October 21, 2010 -- Following HP's $2.4 billion acquisition of 3PAR, speculation continues to run rampant over which storage vendors will be acquired next. Whether it's a good thing or a bad thing from the perspective of end users, further contraction in the industry is inevitable as the largest IT vendors attempt to control the entire IT stack and the pure-play storage vendors plug the gaps in their product lines in order to goose revenue growth.

For the following list of the Top 10 storage acquisition candidates, I factored in the opinions of InfoStor.com readers, which I received after posting "The Top 10 storage acquisitions of 2010." Interestingly, the majority of the reader responses came from channel professionals -- VARs and integrators -- leading me to believe that channel pros are much more interested in mergers and acquisitions than are end users. I also factored in opinions from industry and financial analysts. And topped it off with my own misguided opinions.

(Note: The ticker symbol links below take you to that company's entry page on the InfoStor Market Index, which provides up-to-the-minute info on the company's stock as well as company- and competitor-related news.)

#1 -- Isilon

Isilon scooted to the top of this list because of two recent developments: (a) The company hired Qatalyst Partners to solicit potential acquisition offers. Qatalyst was the advisor to Data Domain when EMC acquired Data Domain, and Qatalyst also shepherded the HP-3PAR acquisition. (b) The NY Post reported on Friday that EMC may be close to acquiring Isilon for about $2 billion (see "EMC in exclusive talks to buy Isilon").

Isilon is known primarily for its strengths in scale-out NAS, but the company recently added support for the iSCSI SAN protocol in its OneFS operating system, enabling both file (NAS) and block (SAN) I/O under a single file system (although Isilon's storage systems do not support the Fibre Channel or FCoE protocols).

In addition to EMC, financial analysts have cited Dell, HP and IBM (and, less likely, Oracle or Cisco) as potential acquirers of Isilon (NSDQ: ISLN).

#2 -- CommVault

I don't understand why CommVault's (NSDQ: CLVT) stock jumped so high during the HP-Dell-3PAR bidding war, but it did. I guess it was because CommVault has been an acquisition speculation darling for years.

CommVault has done a great job stealing revenue from the big four vendors in the data protection space, in part because CommVault's underlying architecture is newer and designed better for rapid enhancements, as evidenced in the recent release of its Simpana 9 software (see "CommVault unveils Simpana 9").

Conventional wisdom on Wall Street has Dell (NSDQ: DELL) as the most likely suitor under the assumption that none of the leading backup/recovery vendors (EMC, Symantec, IBM, CA) would be interested in CommVault, but I'm not so sure about that. Any of those vendors would have to eat some crow if they acquired CommVault, but they might be better positioned for the long haul.

Another possible CommVault suitor: NetApp which, by the way, is the #4 storage software vendor -- ahead of CA and HP (see "The Top 6 storage software vendors").

In addition to the fact that many financial analysts put the company at the top of their storage acquisition target lists, CommVault earns the #2 spot on this list because I think the storage M&A focus is going to shift from hardware to software. And with CommVault, it's hard to argue with low debt, high growth and high margins.

#3, #4 -- Compellent, Xiotech

Isilon, Compellent and Xiotech have all been cited often as potential Dell acquisitions post-3PAR/HP. I doubt it, because Isilon/Compellent/Xiotech are gap pluggers rather than the game changer that 3PAR would have been for Dell. Isilon, Compellent or Xiotech don't give Dell the high-end array technology that would enable Dell to go up against EMC/IBM/Hitachi, although they would be good complements to Dell's EqualLogic line. On the other hand, any one of these disk array vendors would help Dell doff its "Dude, you're getting a Dell" image.

Given their technology differentiators, and the difficulty of being a relatively small player in the contracting disk array market, it's likely that one or both of these vendors will be acquired by someone.

Xiotech, which is not publicly traded and has a nice differentiator with its Intelligent Storage Element (ISE) technology, would be the least expensive buy in this category.


The question with Compellent is whether the company has enough differentiation from what the potential acquirers already have in their portfolios.

#5 -- Permabit

Earlier this year, Permabit introduced an "OEM embeddable" version of its data deduplication software. dubbed Albierio. The company seems to be off to a good start with its OEM strategy, having already racked up reseller deals with BlueArc and Xiotech (see "Data deduplication: Permabit finds success with OEM model").

As the last standing "independent" data deduplication player, Permabit could thrive with its OEM model, but in light of the IBM-Storwize and Dell-Ocarina acquisitions in the data reduction space I think Permabit is an attractive acquisition candidate. The company's focus now is on deduplication for primary storage, but there's no reason Permabit's technology couldn't be used across all storage tiers -- and that could be very attractive for some of the larger storage vendors that have a diverse mix of data reduction solutions in their portfolios. Of course, that assumes that a "one size fits all" approach to data reduction is the way the larger vendors want to go in this space.

#6 - Brocade

IBM has been mentioned most frequently as a potential suitor for Brocade. That might have changed recently with Big Blue's announcement that it plans to acquire Blade Network Technologies (BNT), making it less likely that IBM would go for Brocade.


BNT specializes in blade and rack Ethernet switches, so it's not an overlap with Brocade's business, but the IBM-BNT acquisition will still dampen speculation that IBM will scoop up Brocade. More likely, perhaps, IBM will go after Juniper Networks. (Both Brocade and Juniper are IBM partners and, making it even more interesting, Juniper is a BNT partner.)

If we take IBM out of the Brocade acquirer lineup, that would leave Dell as the most likely acquirer (although Dell is also tight with Juniper). But that goes to the heart of the question of whether Dell will, post-3PAR, try for another disk array vendor (Isilon, Compellent, Xiotech?), software vendor (CommVault?) or surprise everyone and light out into LAN/SANland via Brocade.

Making a Dell acquisition shift toward networking more likely, Dell recently hired a former Cisco exec -- Dario Zamarian -- to run its networking business. Assuming Dell turns to networking in its acquisition spree, it's a 50/50 bet between Brocade and Juniper, according to some Wall Street wags.

In an article posted on MarketWatch (see "Brocade targeted by M&A rumor mill"), Wedbush analyst Kaushik Roy was quoted as saying, "It makes a lot more sense for Dell to buy Brocade than IBM. With Dell, it's a no-brainer. If Dell has half of a brain, they should be taking Brocade out right now."

Then again, if Dell goes after Brocade (NSDQ: BRCD), they could wind up in another crazy bidding war with, say, IBM or Oracle. And Dell's been there, done that.

In addition to IBM and Dell, analysts have cited Oracle and Hitachi as possible acquirers.

#7 -- BlueArc

Hitachi Data Systems (HDS) has been mentioned as a potential suitor for BlueArc, but it doesn't seem to be in HDS' genes to take the acquisitions route. But HDS and BlueArc do have a tight relationship and BlueArc has some attractive technology differentiators.

# 8 -- FalconStor

Only one of our readers mentioned FalconStor as an acquisition target, and it was in the context of the company potentially being acquired by HDS or NEC, but FalconStor would be relatively inexpensive and the company has great technology (although not everybody knows it because FalconStor's software is often sold "under the covers" by its OEMs and resellers).

In addition to getting storage management and data protection software (VTL, CDP, data deduplication, replication, etc.) across a variety of product lines, an acquirer could put the hurt on a lot of competitors because, although a relatively small company, FalconStor's tentacles reach across a lot of vendors via its reseller deals.

The possibility of FalconStor (NSDQ: FALC) being acquired heightened recently with the resignation of ReiJane Huai, the company's CEO. In the wake of the resignation, FalconStor tapped Jim McNiel as interim CEO and president. Some analysts think that McNiel may be more open to acquisition than was Huai.

#9 -- Symantec

The latest rumors regarding Symantec centered on Microsoft as a potential acquirer (see "Is Microsoft Looking to Buy Symantec?" on InfoStor sister site eSecurity Planet).

However, that (unlikely) move would be more for Symantec's security product line, rather than its storage software, and would be in response to Intel's acquisition of McAfee.

Given the breadth of Symantec's product line, and the fact that the company would be expensive and is not a pure-play storage vendor, I put Symantec low on this list. I don't see Symantec being bought any time soon, if only because of the company's huge market cap.

#10 -- NetApp

Rumors about NetApp (NSDQ: NTAP) being acquired have existed as long as the rumors surrounding CommVault and Brocade have, maybe longer. I think the time to buy NetApp is long gone, but due to the persistence of the rumors the company still makes our Top 10 list.

I see NetApp more as an acquirer than a target. The problem with that (for NetApp) is that, if the IT market does contract down to five or six soup-to-nuts vendors, NetApp doesn't have a chance of making that list even if it does dip into its deep piggy bank to make some acquisitions. It's that logic that keeps NetApp on our Top 10 acquisition targets list.

One other company that occasionally comes up in acquisition conversations is Quantum. This would be for the company's data deduplication technology and products. However, an acquirer would also get Quantum's tape business, and I don't think any of the likely acquiring vendors is looking to add tape to their portfolio.

And in the Least-Likely-But-Often-Mentioned category of acquisition targets: EMC. The Wall Street Journal reported last week that Oracle may be eying EMC (see "EMC Shares Rise On Oracle Buyout Rumor"). Sure, that would fill in the Tier-1 storage hole in Oracle's portfolio and, more importantly, give them VMware, but I still think the Oracle-EMC acquisition speculation is ludicrous.

Related blog post:

The Top 10 storage acquisitions of 2010

Data deduplication: Permabit finds success with OEM model

September 20, 2010 -- Following the blockbuster acquisitions of Ocarina by Dell and Storwize by IBM (well, pre-HP/3PAR they were blockbusters), all eyes were re-focused on the data deduplication space (aka data, or capacity, or storage optimization).

Around the same time (early summer), Permabit introduced an OEM version of its data deduplication software, dubbed Albierio, that storage hardware and software vendors can integrate into their products.

Permabit is among only a few "independent" deduplication vendors that are taking the OEM route rather than the acquisition route (although I still say the company is an attractive takeover candidate).

So far, Permabit's strategy seems to be paying off. The company has inked two OEM deals -- with BlueArc about a month ago and, today, Xiotech -- and I understand that Permabit has a few more OEM deals that will be announced over the next couple months.

Permabit claims zero performance degradation with its Albireo deduplication software, and both BlueArc and Xiotech officials cited Albireo's performance, as well as scalability and data "safety" (Albireo doesn't alter data) in announcing their OEM agreements. BlueArc and Xiotech are a good start for Permabit, although it's unclear how long it will take either vendor to integrate Albireo into their systems.

Both the BlueArc and Xiotech deals are focused on data deduplication for primary storage, as opposed to secondary or backup storage. That's where Permabit is currently focused, although Albireo software can be used across all storage tiers, and can be implemented with block, file or unified (block + file) storage.

I recently caught up with Tom Cook, Permabit's president and CEO, to talk about trends in the capacity optimization market.

I asked him about the fact that some of the large storage vendors seem to be embracing multiple capacity optimization techniques for different tiers of storage and, in some cases, carrying a mix of homegrown and OEM'd solutions for data deduplication and/or compression.

"We're still in the very early stages of data optimization, where you have single-function devices such as D2D and VTL appliances," says Cook. "But over time, the big players will try to get to a single data optimization solution that covers all storage tiers."

"Vendors will drive toward total efficiency, where's there's no re-hydration of data," says Cook. "And we believe that the best place to start is on primary storage. It's like New York: 'If you can make it there, you can make it anywhere.'"

Related articles:
Dell to acquire Ocarina for data deduplication
IBM scoops up Storwize

Ocarina: 4 dedupe predictions, and an OEM strategy

June 22, 2010 – A couple of recent news announcements/rumors have focused attention on data reduction (compression/deduplication) for primary storage, also referred to as primary storage optimization, capacity optimization, storage efficiency, and many other terms that may be confusing end users.

First, Permabit announced a software-only version of its data deduplication technology designed for storage hardware and software OEMs (see “Permabit deduplicates primary storage”). If Permabit succeeds in nailing a major OEM deal or two (and the company claims to have at least one in the works) this could be a game changer. For a second opinion on that, see Jeff Boles’ blog post.

Second, the rumors are still swirling around IBM’s alleged negotiations to buy primary storage compression specialist Storwize (see “IBM to acquire Storwize for $140 million?”).

I recently spoke to Murli Thirumale, CEO at Ocarina Networks, which is a Storwize competitor and soon-to-be Permabit competitor. Leading up to a product/strategy announcement that Ocarina made today (see below), Murli offered four predictions on data reduction for primary storage:

--“Every major storage vendor will have primary storage deduplication in their portfolio by 2011, whether by building it or buying it. And by 2012 every major host vendor – including servers and virtualization platforms – will have data reduction built in.”

--“The compression vs. deduplication argument will go away as users realize that you need both. Each technology has advantages/disadvantages for different data types and access patterns.” [Ocarina’s technology combines compression and deduplication.]

--“The industry will eventually have solutions integrated across all tiers of storage, rather than point solutions for different tiers. You shouldn’t have to re-hydrate across tiers or workflows. Once you shrink the data, keep it shrunk.”

--“Deduplication is not a feature; it’s a business. The data deduplication market will exceed $3 billion within five years. And I think it will ramp faster on primary storage than it did in the backup market.”

Which sort of leads into the announcement that Ocarina made today. In a strategy similar to Permabit’s, Ocarina announced a software-only, ‘embeddable’ version of its ECOsystem data reduction technology targeted at OEMs. (Ocarina’s existing data reduction technology is delivered as a software-hardware appliance.)

Ocarina may not be a household name, but note that its partners/resellers include, among others, HP, EMC, Hitachi Data Systems and BlueArc, although Ocarina didn’t make any OEM announcements for its embeddable software today. (It takes a long time to qualify and integrate technology like this into a vendor’s existing stack.)

One differentiator that Ocarina may have is a content-aware data reduction technology with a combination of data compression and deduplication. In addition, Ocarina CEO Thirumale says that the OEM version of the company’s technology can be applied anywhere in the IT infrastructure – from primary storage devices to archiving platforms – and on any type of data (e.g., files or blocks), enabling data to remain in compressed/deduped form throughout its lifecycle. In addition, the software can be implemented in a post-process, in-band or hybrid approach.

In other data deduplication news today, GreenBytes announced its entry into the European market with a variety of reseller partnerships. GreenBytes packages its data deduplication software in inline GB-X appliances that combine solid-state disk (SSD) drives and 2.5-inch SATA drives in what the company refers to as a Hybrid Storage Architecture (HSA). GreenBytes’ block-level deduplication can be used in NAS or SAN environments.

IBM to acquire Storwize for $140 million?

June 14, 2010 (Updated June 15) – Rumors swirled this week around speculation that IBM is in talks to acquire startup Storwize, which specializes in data compression/reduction for primary storage, for a reported $140 million. I wasn’t able to confirm the rumor, but I have no doubt it’s true (the talking part, although not necessarily the acquisition part).

The potential deal makes sense for a number of reasons:

One: Data reduction for primary storage is going to take off big time this year. Heavyweights EMC and, more so, NetApp have capitalized on this trend with data deduplication techniques for data reduction. IBM has data deduplication for secondary storage via its acquisition of Diligent Technologies, but has been out of the picture when it comes to data reduction for primary storage. And buying Storwize (already an IBM partner) will get Big Blue into the game faster than developing the technology on its own.

Although it’s based on compression, not data deduplication, Storwize’s technology (Random Access Compression Engine, or RACE) can be used in conjunction with data deduplication products such as those from EMC and NetApp.

Two: IBM has plenty of experience acquiring Israeli companies. In addition to Diligent, IBM acquired Israel-based XIV in 2008. (Storwize’s R&D facility is in Israel, although the company now has U.S. headquarters.)

Three: IBM has already been reselling Storwize’s STN appliances, and has jointly presented Webcasts with Storwize and produced white papers legitimizing Storwize’s various claims, including the claim that its compression appliances can reduce capacity by 50% to 90% without negatively affecting performance.

Four: It hasn’t been a secret that Storwize has been shopping itself for some time. Getting acquired makes more sense than trying to go it alone in the primary storage optimization market, which will be ruled by the big boys – not startups (just my opinion, but I’m right.)

This acquisition would be interesting on a number of fronts.

--Would it lead to HP acquiring partner Ocarina Networks, a Storwize competitor?

--And let's not forget about other players in the primary storage optimization space. Exar (via its acquisition of Hifn) has data reduction technology that combines data deduplication and compression and handles those functions at the ASIC level. And GreenBytes has inline deduplication ttechnology that can be applied to NAS or SAN environments.

--Could IBM’s offer lead to a bidding war? Perhaps between Big Blue, EMC, NetApp and/or HP?

Further fueling the flames in this market was Permabit’s announcement of its Albireo technology last week (see “Permabit deduplicates primary storage”). Albireo is a software-only deduplication technology designed specifically for primary storage, and the company claims to have a number of design wins, with at least one rumored to be with a big vendor. Who could that be?

This could get even more interesting than last year’s NetApp-EMC bidding war for Data Domain.

Related articles:
Data reduction for primary storage: Benefits and options
Storwize upgrades compression appliances
Infineta unveils deduplication for inter-data-center traffic
GreenByes launches entry-level dedupe appliance
Nexenta debuts ZFS-based in-line deduplication

What is Dedupe 2.0?

September 28, 2009 – Dedupe 2.0 is a term used primarily by Permabit, and it involves extending the benefits of data deduplication across the entire storage environment – not just the traditional application of deduplication on secondary (backup/archive) devices but on primary storage devices as well.

And, yes, it has applicability to cloud storage, whether it’s internal/private or external/hosted. (Then again, what doesn’t have applicability to cloud storage these days?)

If reducing your overall storage costs by 10x or 20x via across-the-board deduplication and capacity optimization sounds appealing, you might want to view our recent Webcast, “Dedupe 2.0: The Benefits of Optimizing Primary and Secondary Storage.” It originally aired last week, but is accessible here.

Jeff Boles, a senior analyst with the Taneja Group research and consulting firm, addresses the IT challenges and issues and Mike Ivanov, vice president of marketing at Permabit, explores solutions.