Let’s face it; many of us are storage junkies. But, we’re not totally to blame for our love of storage. According to analyst firm IDC, data that is housed in today’s enterprise data centers is growing at an astounding 60 percent per year, resulting in 14 exabytes (that’s with 18 zeros) of corporate data by 2011. How many times have you heard, “We’re out of storage. We need to purchase more.”
Organizations are trained to need bigger, better, faster, and more storage. As their appetites swell, they consume and create ever-more data from increasingly diverse sources. And while this phenomenon can be an immense asset, it also spawns new challenges in manageability, usability, and data center resources. These add up to systemic storage problems that can exhaust and bankrupt any IT department.
We’ve entered the age of Big Data; the age of information overload. Now, it’s the demand on storage and retrieval that has become the biggest challenge for IT departments.
In contrast to years past, where information was neatly compartmented, Big Data has become widely distributed, scattered across many sites on different generations of storage devices and equipment brands – some up in the clouds, others down in the basement. While “experts” would have you store it all in one place, that advice is both impractical and flawed.
Impractical, because it’s incredibly difficult to lump all that gargantuan data in one neat little bucket, and flawed because doing so would expose you to catastrophic single points of failure and disruption. Not because the equipment is unreliable, but because the facilities used to house it constantly conspire against you. Having said that, centralized management and coordination of distributed data resources is very doable, particularly when you take a uniform view of storage assets from a hardware-independent perspective.
But like the beginning of any 12-step program, that is only the problem, with a broad answer. In the next post, I’ll discuss new innovations that companies can explore in order to better manage Big Data and specifics they can use in order to cure their storage addiction.