The Petabyte Challenge…

Managing large amounts of data is very challenging technical endeavor, but for many cloud storage providers the volume of data they are tasked with protecting and maintaining is so large that it presents unique challenges.

One of those particularly daunting issues is “silent data corruption.” This topic is discussed by Zetta CTO Jeff Whitehead in a recent blog entry. Whitehead’s excellent description of the problem and how to analyze it includes a calculator to help estimate the probability of random disk failures – this should be required reading for any system administrator or architect of a cloud storage solution. If you’ve built one and this is news to you – you (and your customers) are in trouble…

We included a large excerpt below (Jeff: let us know if you would prefer we take it down):

IT professionals are well aware of many challenges related to scaling storage: capital required to house data, manage backups, data center space, power and cooling. One area many IT professionals haven’t had time to look at, however, is how increasing data footprints translate into increased risk of data loss or data corruption. To put this in context, IDC recently reported that data volumes will increase by a “factor of almost five,” while “total IT budgets worldwide will only grow by a factor of 1.2 and IT staff by a factor of 1.1.” In this context of constraints, being asked to do more with less, without special attention to data risk management, risk inevitably increases.

I believe that many IT professionals and CIO’s will be very surprised to see that while Data Loss (ie, simultaneous drive failures) may not be very probable, Data Corruption (the data on disk is no longer what was originally written out by the application) is shockingly likely, and has caused outages for even some of the most technologically advanced high end environments.

The objective of this blog is to introduce or reintroduce the concept of “Mean Time To Data Loss (MTTDL),” whereby IT professionals, CIOs, and risk managers can create a probabilistic model for evaluating the reliability and probability of data loss for your current environment, and also compare and contrast with how Zetta is advancing the state of the art for cost effective data protection.

MTTDL is a tool, and to be effective one must understand its limitations. The inputs to the model are as follows:

The number of hard drives (data set size/system performance)
The reliability of each hard drive
The probability of reading a given hard drive correctly without error (see prior blog about silent data corruption)
The redundancy encoding of the system
The rebuild rate.
Mean Time to Data Loss is in many respects a best case scenario, because it ignores risks to data integrity such as fire, natural disaster, human error, and other common causes of storage failures. It also ignores autocorrelation¸ or drives failing at the same time due to similar workload, similar manufacturing batches, firmware issues, or the like. Despite these limitations, MTTDL is still one of the better tools for evaluating the data protection features of a storage system.

Update on Carbonite Incident – More Info From CEO…

In response to our recent post regarding Carbonite’s lawsuit against its hardware vendor, we received a comment from Carbonite’s CEO David Friend. The entire comment has been approved and published, but we wanted to make sure that his key points were recognized with a full posting, as opposed to just showing up in the comments of the original.

Key Point 1: Friend notes that the incident happened over a year ago and is “not an ongoing problem.”

Key Point 2: Friend asserts that while approximately 7,500 users were originally affected by the problems, only 54 users actually lost data (and most did not lose all of their data).

Key Point 3: Friend asserts that Carbonite has change its practices and its vendors to increase the reliability of its systems such that, according to the CEO, the chances of failure “are almost nil.”

It seems like a positive indication that Carbonite’s CEO is addressing this issue head on. The questions that remain are: 1) who is ultimately responsible for the problems that occurred, 2) is the new design Carbonite is using sufficiently reliable, and 3) how will this issue affect cloud storage providers and consumers?

Carbonite Loses Data for Thousands of Customers, Sues Vendors

Online backup company Carbonite alerted the public that it had lost data belonging to over 7,500 customers over a number of separate incidents by filing a law suit against a hardware vendor and systems integrator. Carbonite claims that the cloud storage disaster was the result of $3M in faulty equipment provided by Promise Technology Inc. and has brought suit in Suffolk County. The Boston Globe reports that Promise denies any wrongdoing or liability.

Regardless of the outcome, events like this do not bode well for cloud storage providers. Failures, regardless of who is at fault, damage the critical consumer confidence that cloud storage requires to thrive. The impact of this breakdown, from a well-funded (the Globe indicated that Carbonite has raised over $46m) and well-known player in the online backup space, remains to be seen.

Thoughts? Please let us know what you think…

How does Rackspace’s Cloud Files stack up against S3?

We’re very interested in feedback on the newly launched Cloud Files services from Rackspace. The Rackers posted a very interesting blog entry that included performance analysis and cost comparisons that pits their service squarely against Amazon’s Simple Storage Service (S3). If you are using the Cloud Files system, please post a comment and let us (and our readers) know what you think.

Shortly, we will be adding the Rackspace storage service to our directory (just gathering a little info for now)…

NETGEAR Details Embedded Cloud Storage

Earlier this week at the German trade show CeBIT, networking solutions provider Netgear (NASDAQ:NTGR) announced their entrance into the cloud storage space. In an extension of their fast growing network attached storage product, the ReadyNAS, the firm is offering a cloud based data protection service that comes pre-installed on new devices. A spokesperson described it as “the first NAS-Linked Online Disaster Recovery for consumers and SMBs.”

Closer inspection reveals that the service is delivered by existing cloud storage player ElephantDrive. The details of the partnership were not disclosed, but it looks like the accounts will differ dramatically from the regular small business storage accounts profiled in our directory.

Reports of the new service do not yet include reviews or performance metrics, but more information can be found at bMighty and CNET.

HP Upline is now HP Offline, as the service is discontinued

We continue to see new entrants to the cloud storage space, which brand new firms popping up seemingly each week. That said, it is starting to appear that delivering a profitable and effective service is harder than it looks – especially when you’ve got high expectations, as HP did for its Upline offering.

It appears that Upline is now set to be “Offline.” We first heard the story from the Technology Expert Blog, where the team reported that the Upline online storage service would be discontinued. It seems like a rather abrupt shutdown, but HP in indicating that it will refund all charges.

We covered the launch of Upline here back in April of last year. It turns out that there was some rocky performance along the way (as reported by TechCrunch and Enterprise Storage Forum), but we didn’t see this coming from a company that has the resources of HP.

Please let us know if you’ve got more details…

New cloud storage entrants with unique twists…

Today we’re examining two new cloud storage players, each with their own take on what features users really want from the cloud.

The first is ZumoDrive. ZumoDrive is using a combination of the cloud and a drive on your device to give you a more transparent experience for your netbook and iPhone. It’s not synching so your files feel and act local.
Its lets you put any amount of media from your Mac or PC onto your iPhone. It’s in the App Store now. Here’s a link to yesterday’s press announcement.

The second is a company called Attassa, and they are focused on managing email via Outlook. Attassa claims to makes it easy. If you lose the data on your computer or just need to move your email to a new one, they promote one-click “Restore” to get you back up and running. The company plans to continue improving the email experience overall will soon release a service making your entire email archive easier to access and navigate from your mobile device.

We will review both services and ultimately add them to the directory.

Microsoft Opens SkyBox

On Monday, a site at Microsoft was briefly up, pulled down, and changed to a teaser offering of an online backup service for mobile phones. According to Engadget, the service will offer up to 200 MB for transferring contacts, appointments, tasks, photos, videos, music, text messages, and documents for Windows Mobile 6 phones and is “free at this time.”  The page now labels the brand as “My Phone,” although the URL is www.getskybox.com, perhaps a tip of the hat to Sequoia backed DropBox (or was that Dropboks?).

The offering appears to be a direct challenge to Apple’s MobileMe service for synchronizing iPhones and iPod Touch devices to Macs and PCs,  which offers 20 GB of storage for $99 / year.

Today there is a trend toward higher storage densities on smartphones, an increasingly dominant player in the mobile market (Apple), and vicious competition amongst web storage companies, including Microsoft’s own SkyDrive.  Any player hoping to win in this arena will certainly have to make mobile compatibility a key piece of their platform without creating an additional “island” of data.  It remains to be seen how Microsoft will offer additional storage, at what cost, and how it  integrate across their other web offerings.  We’ll be reviewing it here as soon as it becomes publicly available.

Carbonite Caught Posting Bogus Online Backup Reviews

The New York Times reports that online backup provider Carbonite was caught posting positive reviews about its service. The tech columnist for the Times, David Pogue, highlighted a similar discovery around reviews posted on behalf of components firm Belkin (they paid people to post positive or “five-star” reviews), but certainly aimed most of this blog entry at Carbonite.

Pogue’s column was inspired by one of his readers. The reader was a troubled user of Carbonite who chronicles he problems and sleuthing at his own blog in a post titled “Question of Trust”. As Pogue notes, this blogger “is not a user you want to disgruntle.” The comments include a response from the Massachusetts-based company’s CEO pledging to eliminate these reviews (many have already been taken offline by the review publishers) and ensure that no similar problems occur in the future. Also, he notes that at the time they were posted the company was much smaller and did not yet have the appropriate policies in place.

Xdrive: Going, going… Sunset!

After two months of transition, the Xdrive service has officially closed. If you have any doubt, check out their site which simply says “Xdrive is closed”. Fortunately, the AOL team that decided to “sunset” the Xdrive product (along with AOL Pictures and the still nascent BlueString media management platform) gave the user base plenty of notice — they originally leaked the news in July of last year and then made the official announcement back in early November.

In spite of all that warning, it seems safe to assume that thousands of users will only realize that their data is toast now, when it is too late to retrieve the files.