GigaOm Radar for High-Performance Object Storagev3.0

1. Summary

For some time, users have asked for object storage solutions with better performance characteristics. To satisfy such requests, several factors must first be considered:

• Data consolidation: Combining and storing various types of data in a single place can help to minimize the number of storage systems, lower costs, and improve infrastructure efficiency.

• New workloads and applications: Thanks to the cloud and other technology, developers have finally embraced object storage APIs, and both custom and commercial applications now support object storage. Moreover, there is a high demand for object storage for AI/ML and other advanced workflows in which rich metadata can play an important role.

• Better economics at scale: Object storage is typically much more cost-effective than file storage and easier to manage at the petabyte scale. And $/GB is just one aspect; generally, the overall TCO of an object storage solution is better than it is for file and block systems.

• Security: Some features of object stores, such as the object lock API, increase data safety and security against errors and malicious attacks.

• Accessibility: Object stores are easier to access than file or block storage, making it the right target for IoT, AI, analytics, and any workflow that collects and shares large amounts of data or requires parallel and diversified data access.

Many applications find object storage a natural repository for their data because of its scalability and ease of access. However, older object stores were not designed for flash memory, nor were they optimized to deal with very small files (512KB and less). Many vendors are redesigning the backend of their solution to respond to these new needs, but in the meantime, a new generation of fast object stores has become available for these workloads.

These new object stores usually offer a subset of the features of traditional object stores, in particular geo-replication or S3 API compatibility, but they excel in other ways that are even more important for interactive and high-performance workloads, including strong consistency, small file optimization, file-object parity, and features aimed at simplified data ingestion and access with the lowest possible latency. Their design is based on the latest technology: flash memory, persistent memory, and high-speed networks are usually combined with the latest innovations in software optimization. Even though object stores will never provide the performance of block or file storage, it is important to note that they are more secure and easier to manage at scale than the others, offering a good balance among performance, scalability, and TCO.

Maintaining a consistent response time under multiple different workloads is also very important. On the one hand, there are the primary workloads for which these object stores are usually selected, but on the other, it is unusual to find fast object stores serving only a single workload over a long period. Users tend to consolidate additional data and workloads, and multitenancy quickly becomes another important requirement. These solutions typically offer good file storage capabilities, allowing data to be consolidated even further.

Presently, high-performance object stores do not overlap with traditional object stores except for a limited set of use cases. This distinction will change over time because both traditional and high-performance object stores will eventually add the features necessary for parity. The products with the most balanced architecture and the ability to optimize for the latest media will end up in the leading positions.