In the world of data storage, performance and cost are two critical factors to optimize. The performance of your storage hardware has a direct impact on application responsiveness and, in turn, the user experience.
When you can control what data gets placed on which storage device, you can positively impact performance, and at the same time, reduce storage costs that are increasing with growing data volumes.
Every chunk of data stored on your storage infrastructure occupies space and soon fills up available capacity. But not all of the data that gets written to storage is equally important. While there is mission-critical application data that is written to and accessed by databases, there is also a bulk of data generated that is not very frequently accessed, such as snapshots, log files, analytics data, etc.
There could also be specific files (based on their metadata age, type, last modified, author, etc.) that you want to store in a certain storage location.
As an IT/storage administrator, to be able to manually determine what data to place on which storage and to do it in real-time with changing patterns of user access and business conditions is close to impossible. This is when IT teams rely on storage tiering and data placement automation.
Automate Data Placement with Software-Defined Storage
DataCore provides enterprise-class software-defined storage (SDS) solutions that deliver the automation you need to dynamically tier data across different storage classes. The software leverages AI/ML-assisted automation to govern data placement based on business requirements.
DataCore offers three different solutions to choose from based on your specific needs. Each of these uses a unique approach to data tiering and placement helping you eliminate futile manual efforts and accelerate time-to-value.
In all three choices, data is dynamically migrated between storage tiers based on usage patterns and organizational policies.
They ensure your fastest storage devices store only your most important data, and the less relevant data is automatically moved over to secondary storage and other archival tiers.
This allows you to optimize for cost and performance while maximizing the returns from your existing storage investments.
Automated Data Tiering for Block Storage Environments
DataCore SANsymphony is a software-defined storage solution that aggregates resources from disparate storage devices into a virtual storage pool. Within the storage pool, SANsymphony tiers data to the appropriate storage equipment based on how frequently (hot) or infrequently (warm/cold) it is accessed.
Hot data automatically gets placed on the fastest and most performant storage, whereas warm and cold data gets tiered to commodity hardware. SANsymphony uses built-in machine learning algorithms to continuously monitor data access frequencies and dynamically migrate data between tiers.
Typically, hot data are written to the faster flash/SSDs, SAS disks store warm data, and low-cost SATA disks house colder data. If you have faster storage, such as Intel Optane-powered disks in the mix, that will assume the spot of the highest tier, followed by the rest in order of IOPS performance. You simply designate which devices belong in which tier, then sit back, relax and let SANsymphony take over.
Without any manual intervention, SANsymphony will tier data in real-time. The tiering process is automated, fully transparent to the applications, and is continuously happening.
Not just when data gets written to storage initially, but even as data access frequencies change, SANsymphony tracks these pattern changes and dynamically keeps relocating data to the appropriate storage tier as its value changes over time.
Unlike competing alternatives that force an entire volume to be pinned to a tier, SANsymphony distinguishes between blocks within a volume, so only the heavily accessed chunks migrate to faster tiers.
Automated data tiering of hot/warm/cold data
Studies show that typically more than 80% of storage capacity is occupied by dormant data and less than 20% is actively used. You can move all of these dormant blocks to cheaper storage tiers and save the costs of unnecessarily expanding your primary storage.
Data tiering, in addition to other performance-enhancing capabilities in SANsymphony such as high-speed caching, parallel I/O operations, etc., helps to augment storage responsiveness even further
Automated Data Placement for Distributed NAS/File Storage Environments
DataCore vFilO creates a global file system by assimilating a collection of diverse distributed NFS shares under a unified global namespace that makes it easy to access and share files across sites without knowing where they are physically stored. vFilO automates the migration of files between tiers of storage at each site based on administrator-defined policies. These custom policies dictate where data gets placed to align with performance, cost, availability, compliance, and other business objectives.
Using AL/ML-based algorithms, vFilO analyzes patterns, determines the appropriate storage location, and then automatically moves files there. For example, you can set policies to offload large files from a busy NAS (say, *.mp4 files) to a different device and load balance between NAS systems. You may also specify that files meeting a particular criterion relocate to a special file server for legal and compliance reasons.
In addition to data placement between NAS/file servers, vFilO effectively moves seldom-accessed cold/inactive files to lower-cost object storage (on-premises or in the cloud). Since a majority of files are generally infrequently accessed, they can be actively archived, de-duped and compressed for ongoing savings.
Upon access, vFilO rehydrates them automatically back to a primary filer. DataCore Swarm object storage platform provides an excellent secondary storage endpoint for such files.
Automated data placement between NAS/filers and object storage (on-premises and cloud)
A notable highlight of vFilO is that file migration and data placement happen fully transparent to the applications and end-users without disrupting file paths or directory structures.
Since vFilO performs data placement continuously and dynamically, there’s no need for manual intervention, and policies can be adjusted as new objectives arise. vFilO can be a great addition to your IT toolkit as you plan your hybrid cloud strategy.
Policy-Based File Tiering from Windows File Servers to Object Storage
DataCore FileFly software provides a quick and easy means to offload less important file contents from Windows NTFS servers and SMB shares on NetApp and Dell EMC Isilon NAS to lower-cost on-premises object storage.
The destinations include DataCore Swarm and popular S3-compatible cloud storage from AWS, Azure, GCP, Wasabi, and others. Migrated files remain accessible from their original folders and their contents recalled to primary storage when needed.
Using flexible rules and policies, you set which files are migrated and when. You can also schedule migrations outside of peak business hours to reduce I/O contention with production workloads.
For example, you may choose to migrate all files with names ending in *.mp4 (typically large video clips) that have not been accessed in the past 12 months from the Marketing file servers to DataCore Swarm object storage. And schedule the task for Friday nights at 11:30 p.m. when there is very little activity on the servers.
Once FileFly tiers a file’s contents to object storage, it leaves a stub in the on-premises NAS/filer, which serves as a pointer to the new location.
The stub achieves two purposes: 1) it allows users/applications to retain the same file path to access the file when needed 2) it signals backup tools to skip the file for backup, thereby reducing backup time and cost.
File tiering from on-premises NAS/Windows filers to object storage
FileFly includes a data assessment tool and savings calculator that helps you pre-determine how much money you can save by offloading older files to more economical storage.