Experts in the IT industry claim the global volume of electronically stored data is doubling every two years. As businesses increasingly go digital, the growth in the volume of their data is driving greater requirements for efficient and cost-effective storage technologies and data-management tools.
Many organizations are considering outsourcing storage services or cloud storage options—including options for disaster recovery and backup—as a part of their overall storage and data-management strategy. For this process, it’s important to know that the cost of storage services is becoming less dependent on the overall cost of technology needed to store the data and more dependent on the skills and tools to manage it.
Declining Costs in Storing Data
One of the reasons for this shift is the continuing decline in costs for data storage—a decline of as much as 15–20 percent over the last several years. This is the result of several factors, including the following:
- Declining media costs, even for solid-state drives (SSDs), owing at least partly to ever increasing storage-disk densities for all media types
- Better management tools, which drive higher utilization of existing storage resources and make remote storage management easier
- Backup to disk, which reduces requirements for tape handling and off-site storage
- Object storage, which can be more effectively employed for unstructured-data storage and data sets that require multiple online copies
- Requirements for more-elastic capacity, which are more easily delivered owing to the open standards and common APIs for integrating cloud-based options into hybrid solutions
Storage hardware and software have always been major variables affecting the price of outsourced storage services. New technologies that have emerged over the last several years, including SSDs, software-defined storage, hyperconvergence and automated tiering, are driving higher performance and availability but are also making storage and data management more complex. Storage-service providers are experiencing increasing pressure to help manage data, which is an expectation that adds complexity to their role in the overall storage environment.
Prices in the data-storage and data-management market are also influenced by the continuing consolidation of storage vendors and service providers, with a recent one being Dell’s $67 billion acquisition of data storage provider EMC. Meanwhile, many niche players still offer unique solutions at competitive prices, and storage providers are continually entering the marketplace with new technologies as well as more-efficient and more-effective storage-management software.
Changing Storage Strategies
Many organizations have already implemented or are currently implementing an SSD tier in their production storage environments. Although the SSD tier still constitutes a fairly small portion of overall disk storage, it can be effectively used to improve performance for high-end critical business applications and data requiring fast throughput or input/output. The large majority of stored data consists of unstructured files that do not require high-speed access and throughput performance. Typically, more than 60 percent of an organization’s data resides on lower-performance, higher-capacity storage tiers.
Among this unstructured data are large volumes that are near the end of their life cycle and are infrequently accessed by the end user. This data can and should be archived. Although disk is a more expensive option than tape, it has taken the place of tape as a mainstay in most organizations’ backup and disaster-recovery strategies because of requirements for faster and more-immediate access to data copies. The use of tape storage has shifted to an equally important function in the archive space and, therefore, still plays a critical role in most organizations’ overall data-management strategy.
The introduction of better management tools and primary disk-storage deduplication gives enterprises the ability to use storage according to fluctuating performance needs in a more fully automated fashion. Unified storage has introduced simplified “single pane of glass” management capability for storage-area networks (SANs) and network-attached storage (NAS) in a single storage subsystem. Software-defined storage is also changing the landscape for how companies virtualize and store their data. These technologies provide new capabilities that allow more-automated storage management, but they do not resolve the overall planning and decision-making requirements for data management.
Increasing Complexity in Managing Data
The real cost of data storage today, therefore, is not in the storage technologies themselves but in the effort and knowledge required to manage it. This situation is made even more complex by the increasingly stringent requirements for data life-cycle management, heightened country-specific compliance regulations for data-retention periods, increased demand for data retrievals related to e-discovery requests and other legal needs, and greater use of end-to-end encryption.
The goal of any enterprise is to balance the cost of storage with the demand for capacity and performance. Striking this balance requires in-depth knowledge of how an organization is generating, using and storing data from the point of creation to the point of destruction/deletion. Many organizations still lack a data-management policy that mandates when data moves to archival storage and/or when it should be destroyed. Niche storage technologies are being introduced to accommodate these specific needs and functions for data management. These technologies, however, are not always well integrated with the overall end-to-end solution, potentially adding risk and cost to using storage-service providers.
Making a Data Plan
It is important, therefore, that an organization take full responsibility for managing its data and its demand for storage even if it outsources its data storage to a service provider. This effort includes planning and creating an architecture that takes into account new storage technologies and the changing requirements for performance, capacity and availability of data from the application layer.
Large unstructured data sets, for example, should be automatically tiered on the basis of data-usage patterns and performance requirements of the applications. The cost per gigabyte of tape archival storage is orders of magnitude lower than the cost of disk storage. Therefore, mapping the overall storage and data-usage patterns to business requirements for availability as well as the response and retrieval time requirements for data sets can significantly affect the overall cost for a storage service.
Developing an archiving solution that is seamlessly integrated into the overall storage architecture requires planning and a detailed understanding of data life cycles. The architecture and design for the archive environment should remain with the business, not with the storage-service provider, as should the decision about when data is to be archived to lower-performance storage platforms.
Of course, the possibility of using cloud services as part of an overall end-to-end storage solution has introduced an entirely new layer of complexity to the data-storage conundrum. Typically, cloud teams manage end-to-end resources for all infrastructure-technology towers, not just storage. Converged- and hyperconverged-infrastructure solutions designed with built-in automation can simplify the work of moving applications to the cloud by automating and consolidating management of storage, servers and networks. But since public-cloud providers charge for data egress, an organization moving data to or from the cloud needs to determine the usage patterns of a given application before it can accurately forecast its real cost of ownership and management.
If employed properly, many of these new storage technologies can help organizations and storage-service providers more efficiently manage and access their increasing data volume. Data management, however, will remain an ambiguous and hidden cost in sourcing agreements if the scope is not clearly defined, separated from storage management services and priced accordingly. Enterprises need to work closely with their storage-service providers to manage both the data-storage service costs as well as data-management services.
About the Author
Dr. Cindy LaChapelle brings over 25 years of expertise in technical strategy, IT outsourcing (infrastructure) and data center performance assessment to ISG’s clients as a Principal Consultant. Cindy’s in-depth infrastructure review, design and project-management experience includes leading complex organizational assessment and transformation in demanding IT data center environments, and sourcing-strategy development. She also has specialized expertise in data and storage assessment and life-cycle management, as well as backup and recovery and data-protection strategies. Cindy has worked with clients in many different industry and government sectors to help them collect and analyze technical and business data and then to execute, successfully articulate and deliver unique and cost-effective technology solutions to the business.