Object Storage

Scalable and cost-effective storage for long-term data archival and content delivery

Manage unstructured data at scale on-premises with distributed access. Archive and protect your cold data, backups, and media files on S3-enabled object storage.

What is Object Storage?

Object storage, also known as object-based storage, is a data storage architecture designed to handle unstructured data at scale. While block storage breaks data into blocks with unique identifiers, and file systems manage data with a file hierarchy, object storage stores data as objects. Each object acts as a self-contained discrete data repository that has three components: a globally unique identifier that is used to address the object, metadata, and the data payload (or raw data). Metadata contains information about the data payload but can also contain system data such as protection policies or custom metadata.

What is Object Storage | Overview

Data is stored as self-contained and modular units called objects

Objects are stored in a flat organization without any folders or hierarchical structures typically used in a file system. Similar to a key-value pair in databases, object storage uses every object’s unique identifier (key) to refer to the data payload (value). The metadata specific to each object is customizable and can include any user-defined attributes that will help to streamline file search, query, index, and analytics.

Object storage creates namespaces that provide an easy way to access the data stored as objects. Typically, in an object storage cluster, a global namespace forms a single logical access portal for users and applications. Object storage allows data to be accessed through a variety of protocols including S3, HTTP, NFS, and SMB.

According to IDC, 80% of all the data in the world will be unstructured by 2025. This presents the opportunity for all this data to be stored in either file or object storage. Object storage is usually chosen over file storage when data needs to be preserved in an archive for a long period of time – such as cold/inactive data, backup and replicas, compliance data, analytical data, media files, etc. Object storage is significantly more cost-efficient and scalable than block and file storage and is typically implemented on-premises, as a private cloud or on a pay-per-use model as a public cloud service.

Evaluating Object Storage Solutions

By Eric Burgener from IDC and Object Storage Experts from DataCore

Watch the Webinar

Key Capabilities of Object Storage

scalability

Scalability

One of the main reasons organizations use object storage is its unprecedented scalability. Object storage clusters can scale from Terabytes to Exabytes and store billions of files accessed by millions of tenants. Instead of storing files in a file system, files are stored as objects in a flat address space.

By decoupling file management from the low-level block management, every disk in the object storage cluster participates in the namespace and data services are layered on top of that. This level of separation provides excellent manageability at scale. In addition, extensible metadata-based data management simplifies data accessibility. This makes object storage practically limitless in terms of handling any capacity of data.

distributed access

Distributed Access

Object storage enables content access to users and applications via various protocols and access methods including S3/HTTP, NFS, SMB, and REST APIs. Distributed access allows multiple users across distributed locations to access files at the same time.

If any change is needed to be performed by a user at an object level, a new version of the object will be created instead of editing the existing one.  The number of object versions that can be created is often configurable by the administrator. There is also additional policy-based object locking functionality that restricts modification of an object or group of objects for a specific period of time or indefinitely. This level of flexibility for access, editing, and locking enables true distributed access to end users and control for administrators.

custom metadata

Custom Metadata

In object storage solutions, the metadata stores information about the data and is used for data retrieval and governance. Object storage allows users to enrich a file’s metadata with any number of custom attributes.

This simplifies content search, indexing, and analytics. Metadata can also be used to set object-level or bucket-level policies that define how data should be protected and stored.

data durability

Data Durability

While data availability refers to system uptime, data durability is focused on ensuring long-term data protection. Object storage solutions offer strong data protection capabilities to ensure data durability and eliminate single points of failure.

  • Replication of data to one or more nodes ensures data copies are synchronously or asynchronously replicated within the system or externally to a DR (disaster recovery) site. If there is an issue, the object storage system will automatically create additional copies based upon the policy defined to maintain data redundancy. Data copies are also recovered from the DR site in the event of a site failure.
  • Erasure coding combines data with parity information and then segments and distributes it across the object storage cluster. In the event of data loss, data is immediately rebuilt from erasure-coded segments and restored to its original state.
immutability

Immutability

This is the capability of the object storage platform to make objects immutable, i.e., not able to be deleted or modified. Immutability helps preserve records and maintain data integrity. When a user tries to edit a file, a new object version gets created and the original object is preserved as-is.

In case of a threat vector (such as a ransomware attack) or accidental deletion or overwrite of data, administrators can easily roll back to the original data. Compliance regulations also demand data is stored on non-erasable and non-rewritable media and legal holds be applied to files for specific periods of time. Immutability helps meet all these requirements and makes object storage WORM (Write-Once-Read-Many)-compliant.

intelligent data management

Intelligent Data Management

One of the primary use cases for object storage is as an economical archive; however, it is more than “cheap and deep” storage. Object storage can help manage the data lifecycle from creation, change, deletion, access, collaboration, and protection. Object storage supports creating copies of data through replication, locking and encrypting files, protecting against deletion, applying access controls, searching files with rich metadata, integrating with applications through REST APIs, and more.

Some object storage solutions allow content streaming directly from the storage layer without having to download it to a local repository. An intuitive user interface usually serves as the content portal for users to access files over the internet. This expands the use case of object storage from an archive to a full-fledged content management and delivery platform.

multi-tenancy

Multi-Tenancy

A multi-tenant object storage architecture can be used to share resources across a cluster with multiple tenants (internal users or external subscribers). This is a popular implementation by service providers who are hosting storage resources for their clients from a centralized and shared storage cluster – either in a public cloud or private cloud. Tenants are given different levels of access permissions and controls to access data based on their level of subscription. Even within an organization, when different departments need to store and protect their data based on specific policies, they can adopt a multi-tenant deployment within their data center.

Key characteristics of object storage multi-tenancy include:

  • Integration with identity management systems such as AD and LDAP systems to implement access controls and security policies
  • Capacity usage quotas and metering to provide the ability to define and manage utilization across different tenants
  • Auditing and reporting capabilities to help service providers with billing and account management
  • End-user self-servicing options to allow tenant-level management of data with granular access controls, user permissions, quotas, etc.

Object Storage Performance Benchmarking

Utilizing the Science and Technology Facilities Council’s
Super Data Cluster Environment

DOWNLOAD WHITE PAPER

Popular Industry Use Cases for Object Storage

media and entertainment solutions

Media & Entertainment

A content-focused platform enabling secure file distribution, access and tape replacement

high performance computing

High-Performance Computing

Multi-tenant S3 storage that streamlines collaboration and accelerates time to discovery

healthcare data storage solutions

Healthcare

Scalable medical image archive and patient record management

enterprise IT

Enterprise IT

Secondary storage platform to eliminate storage silos and manage unstructured data growth

cloud storage solutions

Cloud Storage

Operational and workflow efficiencies for private, public and hybrid cloud data storage

government data storage solutions

Government

Safeguard data, ensure data integrity, and meet compliance requirements

Differences Between Block, File, and Object Storage

Let’s look at some of the key differences between the architecture, characteristics, and applicability of block storage, file storage, and object storage.

Block StorageFile StorageObject Storage
:Best suited forBlock Storage:Highly transactional data, low-latency applications, databases, and VM workloadsFile Storage:Distributed file sharing and collaboration, scale-out file systemObject Storage:Distributed content access, delivery, and archive
:How data is storedBlock Storage:Data broken into blocks within disk tracks and sectorsFile Storage:Files organized in folders with a directory/pathObject Storage:Self-contained objects
:Data StructureBlock Storage:Volume, LUNFile Storage:HierarchicalObject Storage:Key/Value
:Popular protocols used for data accessBlock Storage:iSCSI and Fibre ChannelFile Storage:NFS and SMBObject Storage:S3, HTTP, NFS, SMB, REST APIs
:Metadata supportBlock Storage:NoFile Storage:Yes, fixed file system metadata. Stored separately from the file.Object Storage:Yes, custom metadata. Stored with the object itself.
:Namespace supportBlock Storage:NoFile Storage:YesObject Storage:Yes
:PerformanceBlock Storage:Very highFile Storage:HighObject Storage:High (optimized for throughput)
:ScalabilityBlock Storage:Comparatively lowFile Storage:HighObject Storage:Very high
:CostBlock Storage:Very highFile Storage:HighObject Storage:Low

Why DataCore Swarm Object Storage

DataCore Swarm provides an on-premises object storage solution that radically simplifies the ability to manage, store, and protect data while allowing S3/HTTP/NFS/SMB access to any application, device, or end-user. Swarm transforms your data archive into a flexible and immediately accessible content library that enables remote workflows, on-demand access, and massive scalability.

Swarm provides a platform for data protection, management, organization, and search at enterprise scale. You no longer need to migrate data into disparate solutions for long-term preservation, delivery, and analysis. Consolidate all files on Swarm, find the data you are looking for quickly, and reduce total cost of ownership by continuously evolving hardware and optimizing resources.

Consumers

  • End Users
  • Application & Web Services
  • Devices

Access Methods

  • S3/HTTP
  • NFS*
  • SMB**

Operation & Insights

  • Usage Auditing,
    Metering & Quotas
  • Identity
    Management
  • Delegation &
    Self-service
  • Historical &
    Real-time Charts
  • Health &
    Performance Graphs
  • Orchestration

Data & Infrastructure Management Services

Command & Control

  • Console
  • REST API
  • Customizable
    Metadata
  • Granular
    Administration

Any Mix Of Storage Media, Any Mix Of X86 Servers

  • HDD
  • SSD

*NFS ACCESS IS SUPPORTED THROUGH SWARMFS. **SMB ACCESS IS SUPPORTED THROUGH DATACORE FILEFLY.

Get Started with Swarm, Software-Defined Object Storage