Redis's rich set of data types allows for easy and fast experimentation of data-based algorithms and approaches on information. In my own experience with Redis, this ability to quickly model and use solutions is based on the characteristics of the different data structures of Redis and the flexibility in defining the structure and syntax of the keys. I was impressed and excited to be able to name a chunk of malleable data and to relate this name with other keys through the naming semantics of the key. This is a great feature of Redis that is sometimes underappreciated as to how powerful and useful a tool it can be in developing and understanding your data.

I first started experimenting with Redis in 2011 as a metadata and systems librarian at Colorado College at the base of the Pikes Peak Mountain in Colorado. Most libraries around the world store and structure their bibliographic data in a somewhat surprisingly durable binary format called, MAachine-Readable Cataloging (MARC), substantially developed in the late 1960s by Henriette Avram of the United States Library of Congress. The current version, MARC 21, is officially supported by the Library of Congress (however, it is in the process of replacing MARC with a new RDF-based linked data vocabulary called BIBFRAME). MARC21 initially encoded information about the books on the library's shelves and has been extended to support e-books available for checkout; video, music, and audio formats; physical formats such as CDs, Blu-ray discs, and online streaming formats; and academic libraries. In fact, an increasingly large percentage of its budget is devoted to the purchase of journal articles through online publishers and electronic-content vendors.

The MARC format is made up of both fixed length and variable-length fields numbered in the three-digit range of 001–999, which in turn can have either character data or subfields with data. In addition, each field can have up to two indicators that modify the meaning of the field. Two of the most common and important MARC fields are the 100 Main Entry – Personal Name field and the 245 Title Statement field. Here is an example from David Foster Wallace's book Infinite Jest:

=100  1\$aWallace, David Foster
=245  10$aInfinite jest :$ba novel$cDavid Foster Wallace

To use this MARC data in Redis, each MARC record was a hash key modeled as marc:{counter} with the counter being a global incremental counter. Each MARC field is a hash with the key modeled as marc:{counter}:{field}. As some MARC fields are repeatable with different information, the hash key would include a global counter such as marc:{counter}:{field}:{field-counter}. Simply storing these two fields would result in the following six Redis commands:

127.0.0.1> INCR marc
(integer 1)
127.0.0.1:6379> INCR marc:1:100 
(integer 1)
127.0.0.1> HSET marc:1:100:1 a "Wallace, David Foster"
OK
127.0.0.1:6379> INCR marc:1:245 
(integer) 1
127.0.0.1:6379> HMSET marc:1:245:1 a "Infinite jest :" b "a novel" 
c "David Foster Wallace"
OK
127.0.0.1:6379> HGETALL marc:1:245:1
1) "a"
2) "Infinite jest :"
3) "b"
4) "a novel"
5) "c"
6) "David Foster Wallace"

This key structure in Redis looks like the following:

MARC in Redis

The storage of MARC data in Redis can be accomplished with just a single Redis data type, a hash, along with a consistent key syntax structure. To improve the usability of this bibliographic data in Redis and to realize a very common use case of retrieving library data as a list of records sorted alphanumerically by title and author name (in library parlance two access points) is also accomplishable with other Redis data types such as lists or sorted sets.

Representing MARC fields and subfields in Redis by using hashes and lists was informative. Further, I wanted to see if Redis could handle other types of book and material metadata models that were being put forward as replacements for MARC. The Functionality Requirements for Bibliographic Record, or FRBR, was a document that put forward an alternative to MARC and was based on entity-relationship (ER) models. The FRBR ER model contained groups of properties that were categorized according to abstraction. The most abstract is the Work class, which represents the most general properties to uniquely identify a creative artifact with such information as titles, authors, and subjects.

The Expression class is made of properties such as edition and translations with a defined relationship to the parent Work. Manifestations and Items are the final two FRBR classes, capturing more specific data where Item is a physical object that is a specific instance of a more general Manifestation.

With few actual systems or technologies that implement an FRBR model for library data, Redis offers a way to test such a model with actual data. Using existing mappings of MARC data to FRBR's Work, Expression, Manifestation, and Item, the MARC 100 and 245 fields from the above would be mapped to an FRBR Work in Redis as shown by these examples of using the Redis command-line tool, redis-cli, to connect to a Redis instance:

127.0.0.1:6379> HMSET frbr:work:1 title "Infinite Jest" "created by" "David Foster Wallace"
OK

This new work, frbr:work:1 can be associated with the remaining classes with the following Redis keys and hashes:

127.0.0.1:6379> HMSET frbr:expression:1 date 1996 "realization of" frbr:work:1
OK
127.0.0.1:6379> HMSET frbr:manifestation:1 publisher "Little, Brown and Company" "physical embodiment of" frbr:expression:1
OK
127.0.0.1:6379> HMSET frbr:item:1 'exemplar of' frbr:manifestation:1 identifier 33027005910579
OK

In the previous example for Expression, a specific date is captured along with a relationship back to frbr:work:1 through the realization of a property. Similarly, the frbr:manifestation:1 hash has two fields; a publisher, and the physical embodiment of. The physical embodiment of field's value is the frbr:expression:1 key that links the Manifestation back to the Expression. Finally the frbr:item:1 hash has a barcode identifier property and a relationship key back to the frbr:manifestation:1 hash.

In both the MARC and FRBR experiments, the Redis hash data structure provided the base representation for the entity. This strategy starts to fail when there can be more than one value for a specific property, such as when representing multiple authors of a work. The first attempt to solve this problem for those properties with multiple values is by creating a counter for each MARC field as outlined above. For example, the MARC 856 field – Electronic Location and Access – stores the URL for e-books or other material that has a network-resolvable URL. If we want to add two URLs to the preceding MARC example, such as a link to the book in Google Books and a wiki on the book, the Redis commands would be as follows:

127.0.0.1:6379> INCR global:marc:1:856
(integer) 1
127.0.0.1:6379> HMSET marc:1:856:1 ind1 4 ind2 1 u https://books.google.com/books?id=Nhe2yvx6hP8C
OK
127.0.0.1:6379> HMSET marc:1:856:2 ind1 4 ind2 2 u http://infinitejest.wallacewiki.com/
OK

This naming approach for the MARC keys meets the requirement for repeating MARC fields, but how can we support the edge case wherein a single MARC field has multiple, repeating subfields? The first pass to solve this problem may be to store a string with some delimiter between each subfield as the value for a particular filed in the MARC. This would require additional parsing on the client side to extract all the different subfields, and we would lose any additional advantages that Redis may provide if these multiple subfields were stored directly in Redis. The second approach to solving the MARC field with multiple subfields in a MARC field would be to further expand the Redis key syntax and use a list or some other data structure as value for each subfield key. Expanding the MARC 856 example, if we wanted to add a second e-book URL, maybe a URL to the Amazon Kindle version, it would look like the following in Redis:

127.0.0.1:6379> LPUSH marc:1:856:1:u https://books.google.com/books?id=Nhe2yvx6hP8C http://www.amazon.com/Infinite-Jest-David-Foster-Wallace/
(integer) 2
127.0.0.1:6379> HSET marc:1:856:1 u marc:1:856:1:u
(integer) 0

Storing multiple subfields in a Redis list works well, but what if I don't want any duplicate values in a MARC field's subfields? This can be easily solved by the use of Redis's set data type, which, by definition, only contains unique values. The use of sets for the subfield values seems like a good solution, but it fails, if we need to keep the ordering of the values in the subfield.

Fortunately, Redis's sorted set data type fits our use case admirably by ensuring a collection of unique subfield values with no duplications, and finally maintaining, the subfield ordering. The resulting Redis commands for storing the URLs of a book in the MARC 856 field would look the following:

127.0.0.1:6379> DEL marc:1:856:1:u
(integer) 1
127.0.0.1:6379> ZADD marc:1:856:1:u 1
https://books.google.com/books?id=Nhe2yvx6hP8C 2
http://www.amazon.com/Infinite-Jest-David-Foster-Wallace/
(integer) 2
127.0.0.1:6379> ZRANGE marc:1:856:1:u 0 -1 WITHSCORES
1) "https://books.google.com/books?id=Nhe2yvx6hP8C"
2) "1"
3) "http://www.amazon.com/Infinite-Jest-David-Foster-Wallace/"
4) "2"

In this example, we examined how to represent a legacy format for library data called MARC, and how MARC's fields and subfields data can be stored in Redis by using hashes, and how the storing of subfields changes as more requirements are met, moving from storing subfields first as Redis lists, followed by sets, and finally finishing by using the sorted set data type. This iterative experimentation hopefully illustrates an important reason for using Redis, namely the ability to quickly test out different methods of storing data and how the characteristics of different Redis data types such as hashes, lists, sets, and sorted sets can be used to represent both the data and some of the requirements for storing and accessing this data.

A very popular use pattern for Redis is as an in-memory cache for web applications. Redis is available as a caching option for popular web frameworks such as Django, Ruby-on-Rails, Node.js, and Flask. As a popular caching technology Redis excels in web applications for storing new data while evicting stale data. For web applications, the cached data can range from single HTML character strings, widgets, and elements to entire web pages and websites.

By utilizing Redis's ability to set an expiration time on a key, one of Redis' popular caching strategies called Less Recently Used (LRU) is robust enough to handle even the largest web properties, with the most popular content remaining in cache but stale and little-used data being evicted from the data store. This caching use case doesn't assume that the original web element or page is generated from the data in Redis; most likely, the web content was dynamically generated from other sources of data with Redis, in this use pattern, and operates as an excellent web caching layer in this setup.

The second popular use pattern for Redis is for the metric storage of such quantitative data such as web page usage and user behavior on gamer leaderboards. Using bit operations on strings, Redis very efficiently stores binary information on a particular characteristic. Usage for a website could be stored with a key constructed from a date such as page-usage:2016-11-01, which has a string attached with a bit flipped to 1 the first time a web page is accessed by a user.

The daily usage for the website for November 1 can be obtained through a simple BITCOUNT Redis command on the page-usage:2016-11-01 key. In a 2011 blog post, individuals at a start-up named Spool explain in detail how they use bitmaps and Redis bit operations to store the user activity on their website with this design pattern.

The third popular Redis use pattern is as communication layer between different systems through a publish/subscribe (pub/sub for short) model, where one can post messages to one or more channels that can be acted upon by other systems that have subscribed to or are listening to that channel for incoming messages.

Typically, publishers do not need to know the specific subscribers to send messages to them (say in a point-to-point messaging model); only the message contents and what channel to send the message should be known. Similarly, a subscriber does not need to know individual publishers, only the channel to receive messages. The pub/sub pattern is nice because it scales easily, and the publishers and subscribers can be very different programs and systems.

As an active open-source project, Redis adds new functionality and improvements that may solve a problem that you or someone in your organization decided it wasn't suited for in the past. Optimizing the use of such a valuable and functional tool as Redis means understanding its recent history and keeping current with new functionality being developed and tested for inclusion in the latest stable version of Redis. Redis follows a common semantic versioning pattern of major.minor.patchlevel with a minor even number denoting a stable version and an odd minor number an unstable branch.

For example, the Redis 2.8.9 release introduced two of the more significant improvements, namely the HyperLogLog, a highly efficient data structure for a population estimate and of unique elements, and the new ZRANGEBYLEX, ZLEXCOUNT, and ZREMRANGEBYLEX commands for sorted sets. Both these are improvements that will be discussed at length in Chapter 2, Advanced Key Management and Data Structures. Redis Cluster – released for production use in early 2015 with Redis version 3.0 – is one of most important additions to the Redis ecosystem, which we will go over in much more detail in Chapter 6, Scaling with Redis Cluster and Sentinel.

For the next major release Redis added Geographic Information Systems (GIS) commands and modified sorted sets along with new Lua scripting support for Redis Cluster and a new Lua debugger in Redis version 3.2. To visualize the rate of change to the Redis code base, the following graphic shows the rate of change in the Redis code base during the Redis 2.x series to Redis version 3.0.

Be aware of the dynamic nature of Redis development when asking yourself, why Redis? The limitations that you thought Redis had might no longer be the case and as you continue to grow your knowledge and improve your skills in mastering Redis, keeping up with Redis changes should a critical priority as you improve your existing technology and build new and exciting opportunities for the future.

The decision as to whether Redis is the correct choice for a new project or to solve a data problem you might be experiencing really depends on the nature of your data and what you're trying to accomplish with your project. Redis, unlike relational databases or NoSQL document stores, does not require you to structure your data first before using it. Redis provides a direct, more algorithmic manipulation of your data through the use of a variety of data structures such as lists, hashes, sets, and sorted sets. Even if Redis is not your final choice, the exercise of breaking down your data into these data structures will help deepen the context and the analysis of the issue that you're trying to solve. A detailed example of such experimentation was given while representing a legacy library standard called MARC in the basic Redis hashes, lists, sets, and sorted sets. We then briefly reviewed three popular design patterns for using Redis as a web cache, Redis as the backend for a gamer leaderboard, and Redis used as a publish/subscribe messaging system. We finish this chapter by illustrating some recent changes to Redis that expand the types of problems that Redis can be the primary data solution that in the past traditional SQL database or other NoSQL technologies may have been adopted instead.

In the next chapter, we are going to first examine Redis keys and the importance of organizing these keys with a Redis key schema generated either through a Redis object mapper or through manual documentation. Chapter 2 then introduces the Big O notation, followed by a systematic review of the basic Redis data structures and commands based on time complexity measures, Chapter 2 finishes with an introduction to some of the newer data structures and commands, including bitstrings and HyperLogLog.