Modern, performance-oriented applications rely on in-memory databases to provide fast and scalable data access for real-time data processing scenarios, such as caching, session management, gaming, fraud detection, leaderboards, real-time analytics, geospatial indexing, ride-sharing, social media, and streaming applications.

Redis is the world’s most popular and widely used in-memory data store. It is an open-source, key-value database that supports various data types, such as strings, lists, sets, hashes, bitmaps, and streams. Redis can also perform atomic operations, transactions, pub/sub messaging, scripting, and geospatial indexing on its data. Redis is often used as a cache, database, or message broker for applications that require fast and scalable data access.

However, Redis was designed over a decade ago, and its single-threaded architecture makes it difficult to scale deployments without compromising performance or reliability. Scaling and maintaining Redis clusters is complicated and time-consuming, and if you don’t manage your Redis cluster properly, you may lose data, which can affect user experience or cause a site outage.

That’s why you should consider using Dragonfly, a modern in-memory database that works as a drop-in replacement for Redis and Memcached. It builds on the success of its predecessors while improving performance and reliability by working with a multi-threaded architecture that allows for more efficient vertical scaling so you can avoid the hassle of horizontal scaling and cluster management while still enjoying the benefits of scaling your website.

But do I mean by all this? In this article, let us look at the main features of Dragonfly and how it compares with the open-source version of Redis on benchmarking tests for throughput, latency, and memory usage.

Redis: A great solution for older workloads

Redis is a single-threaded database, which means it can only handle one task at a time. It doesn’t matter how many threads the machine it runs on has, it can’t use them to do multiple things simultaneously. While Redis can be scaled vertically by increasing the memory and processor speed of the hardware it runs on, the performance improvement is limited. As you can imagine, adding more powerful multi-thread processors to a single-threaded process won’t reap any benefits.

The only way to overcome this limitation is to scale horizontally i.e. by adding more Redis servers, each running a single-threaded Redis process and managing them as a cluster. Redis clusters are notoriously hard to set up and maintain. Unhealthy nodes can’t be easily replaced and must be manually configured, snapshotting must be configured on each node separately and they add complexity to your infrastructure meaning you may need specialized DevOps staff to run and maintain your cluster.

To make matters worse, Redis can run out of memory and crash causing you to lose all your data. Redis needs to run many background processes to maintain a healthy state. However, the workload often becomes too much for its single CPU core to handle causing the database to crash. If this happens your applications may go down or there may be other serious impacts on user experience.

Dragonfly: A modern in-memory database that scales vertically

Dragonfly is a multi-threaded in-memory database that can use the full potential of modern processors that have potentially hundreds of threads and do multiple things at the same time. It can handle millions of operations per second and terabyte-sized workloads on a single instance. Dragonfly eliminates the need for horizontal scaling and cluster management by allowing efficient vertical scaling on a single machine.

Let us take a look at some of the differences between Redis and Dragonfly in order to appreciate what Dragonfly can achieve.

Memory Consumption: Redis uses more memory than Dragonfly for storing data because it uses more memory for internal structures and metadata.

Dragonfly uses less memory than Redis for storing data because it uses new data structures such as dashtable and denseSet, which reduce memory overhead. It uses smarter algorithms and data structures that reduce memory usage and improve snapshotting speed.

Saving to disk: Redis also takes longer to save data to disk, because it uses more disk space and performs more I/O operations.

Dragonfly saves data to disk faster than Redis, because it uses less disk space and performs fewer I/O operations.

Reliability: Redis can run out of memory and crash if it cannot handle the background processes or the incoming requests, which can cause data loss or downtime for the applications that rely on Redis.

Dragonfly is resilient to memory pressure and crashes, which means it can handle situations where the memory is full or the database is overloaded. Dragonfly can avoid these problems by using techniques such as compression, eviction, and throttling, which can reduce memory usage, remove unnecessary data, or slow down requests.

In addition, Dragonfly also supports more data types than Redis, such as documents, graphs, spatial, search, time series, vectors, etc., which offer more features and capabilities for different use cases and applications.

For example, documents are JSON-like objects that can store complex and nested data; graphs are collections of nodes and edges that can represent relationships and networks; spatial are geometric shapes that can support spatial queries and operations; search is inverted indexes that can support full-text search and ranking; time series are sequences of values that can support time-based analysis and aggregation; vectors are arrays of numbers that can support similarity search and machine learning. These data types are not supported by Redis natively, but they can be added with modules.

Benchmarking tests: Throughput

Throughput is the measure of how many requests per second an in-memory database can handle. Higher throughput means better performance and scalability. To compare the throughput of Redis and Dragonfly let's take a look at the results by Memtier_benchmark, a tool developed by Redis Labs that generates NoSQL traffic and measures the performance of the database. The benchmark runs on a single instance of Redis and Dragonfly with different numbers of concurrent clients and different data sizes. The results are shown in the following graphs.

As you can see, Dragonfly beats Redis by a wide margin, achieving up to 30 times higher throughput than Redis. This is because Dragonfly can use all the available cores on the machine while Redis can only use one core. As the number of concurrent clients increases, Redis quickly reaches its limit and cannot handle more requests, while Dragonfly can scale up and handle more load.

Benchmarking tests: Latency

Latency is the measure of how long it takes for an in-memory database to respond to a request. Lower latency means faster response time and better user experience. To compare the latency of Redis and Dragonfly let's look at the results of Memtier_benchmark again, but this time, let's look at the average response time of the database for different numbers of concurrent clients and different data sizes. The results are shown in the following graphs.

As you can see, Dragonfly has much lower latency than Redis, achieving a faster response time than Redis. This is because Dragonfly can handle more requests in parallel while Redis can only handle one request at a time. As the number of concurrent clients increases, Redis becomes slower and slower, while Dragonfly remains fast and consistent.

Benchmarking tests: Memory usage

Memory usage is the measure of how much memory an in-memory database consumes to store its data. Lower memory usage means better resource efficiency and lower cost. To compare the memory usage of Redis and Dragonfly let's look at the results from Memtier_benchmark again, to find the peak resident set size (RSS) of the database process for different numbers of concurrent clients and different data sizes. The results are shown in the following graphs.

As you can see, Dragonfly uses much less memory than Redis, achieving significantly lower memory usage than Redis. This is because Dragonfly uses smarter data structures and algorithms that reduce memory overhead, while Redis uses more memory for internal structures and metadata. As the number of concurrent clients increases, both databases use more memory, but Dragonfly still uses much less than Redis.

Conclusion: Why choose Dragonfly over Redis?

The benchmarking tests show that Dragonfly is better than Redis in every aspect. It can handle more requests per second with lower latency and lower memory usage than Redis on a single instance without any cluster management or sharding complexity. It can also support more data types and features than Redis, such as documents, graphs, spatial, search, time series, and vectors. It can also be extended with modules that provide additional functionality or custom data structures.

If you are looking for a fast, scalable, and reliable in-memory data store for your cloud-native applications, you should give Dragonfly a try. You can easily install it on your machine or use it as a service on the cloud. You can also migrate your existing Redis or Memcached applications to Dragonfly without any code changes. You will be amazed by how much better Dragonfly performs than Redis and how much easier it is to use and maintain.

Dragonfly is an open-source project that welcomes contributions from the community. If you are interested in Dragonfly, you can visit their website, read their documentation, or check out their GitHub page.