Redis

It is single threaded

Types of Redis Architecture

The three main types of Redis Architecture are

• Redis Standalone
• Redis Sentinel
• Redis Cluster (Clustering is a way to share data automatically across multiple cluster nodes (Horizontal Scaling). The cluster will be able to continue operations when some nodes fail or not able to communicate with each other.)

Redis Sentinel

The Redis Sentinel comes up with a master-slave architecture. With this architecture, we will be able to avoid the Single point of failure which was a major concern with the Redis Standalone. Additionally, it comes up with other sets of features, which are

• Monitoring — constantly checking whether the Master and slave instances are working properly or not.
• Notification — Notify systems in case of failure of instances.
• Automatic Failover — In case of a master node failure, the slave node will be promoted to master.
• Configuration Provider — Sentinel nodes also serve as a point of discovery of the current main Redis instance.

Redis server can be run in two modes:

• Master Mode (Redis Master)
• Slave Mode (Redis Slave or Redis Replica)

We can configure which mode to write and read from. It is recommend to serves writes through Redis Master and reads through Redis Slaves.

Redis Master does replicate writes to one or more Redis Slaves. The master-slave replication is done asynchronously.

Redis is AP ( Availability and Partition Tolerance.) system.

What happens when Redis Master receives write request from Client:

• It does acknowledge to Client.
• Redis Master replicates the write request to 1 or more slaves. (Depends on Replication factor).

Here you can see, Redis Master does not wait for replication to be completed on slaves and does acknowledgment to client immediately.

Now lets us assume, Redis Master acknowledged to client and then got crashed. Now one of the Redis Slave (that did not receive the write) will get promoted to Redis Master, loosing the write forever.

In Redis, the automatic promotion of a slave (now called a replica) to master upon the master’s failure does not happen automatically by default. However, you can achieve automatic failover using Redis Sentinel, which is designed to monitor your Redis servers and handle automatic promotion when the master fails.

Redis Persistence Models

• No persistence
• RDB Files: The RDB persistence performs point-in-time snapshots of your dataset at specified intervals.
• AOF (Append Only File):The AOF persistence logs every write operation the server receives that will be played again at server startup, reconstructing the original dataset.

How redis doing snapshot with single thread

Redis leverages forking and copy-on-write to enable efficient data persistence. Forking creates a new process (child) that shares memory with the original (parent) process. Redis, which manages large amounts of memory, uses this mechanism to snapshot data without consuming additional memory unless changes are made. Through copy-on-write, memory pages are only duplicated when modified, allowing the child process to work with a consistent snapshot while keeping memory usage low. This approach enables Redis to capture snapshots of gigabytes of memory quickly and efficiently.

• Atomic Operations: Every Redis command is atomic, meaning that when a command is executing, other commands cannot interrupt it. This ensures data correctness, even with multiple clients connected to the Redis server.

  • This atomicity is especially beneficial for operations like incrementing values, as it avoids concurrency issues where multiple clients trying to increment a value simultaneously could lead to incorrect results.

• In-Memory Data Storage: Redis stores data in memory, making it extremely fast for read and write operations. This is why Redis is often used as a cache.

  • However, Redis also offers configurable persistence options to prevent data loss in case of a crash. These options include:
    • Periodic Disk Dumping: Data is periodically written to disk without deleting it from memory. Upon restarting, Redis loads the last dump.
    • Write-Ahead Logging (AOF): Every update command is logged to an append-only file, allowing for data reconstruction.
    • Asynchronous Replication: Data is replicated to another Redis server.

• Single-Threaded Event Loop: Redis utilizes a single-threaded event loop for handling concurrent client requests, unlike multi-threaded approaches commonly used in databases like MySQL and PostgreSQL.

  • Redis’s Approach: Redis leverages the fact that network I/O operations (like reading data from a socket) are generally slow compared to in-memory operations. It uses IO multiplexing to efficiently monitor multiple sockets and only reads data when it’s available, avoiding unnecessary blocking.
    • This approach allows Redis to handle many concurrent connections on a single thread, as it spends minimal time waiting for I/O operations to complete.
  • Speed and Simplicity: The single-threaded model, combined with in-memory data storage, makes Redis extremely fast. By avoiding multi-threading complexities, Redis also maintains code simplicity and reduces the risk of concurrency-related bugs.

Streams

A Redis stream is a data structure that acts like an append-only log but also implements several operations to overcome some of the limits of a typical append-only log. These include random access in O(1) time

Redis generates a unique ID for each stream entry. You can use these IDs to retrieve their associated entries later or to read and process all subsequent entries in the stream.

In nutshell → we can add data to redis and read from it but it not pub sub we manully need to read from it, data are in ordered

• Data in streams is stored persistently
• Streams support automatic trimming (using MAXLEN), allowing you to control memory usage by removing old messages.

Consumer group

Redis Streams with Consumer Groups allow multiple consumers to share the responsibility of processing messages from a single stream. Each message is delivered to only one consumer in the group, ensuring parallel processing and scalability. If a consumer fails to process a message, another consumer can claim and process the unacknowledged message, ensuring fault tolerance and no message loss. it is similar to rabbitmq or message broker behaviour

CMD

 
XADD mystream * event_type "user_signup" user_id 12345 -> add the data to stream
 
XRANGE mystream start-id end-id` -> to read data from stream
 
XGROUP CREATE mystream mygroup 0 ->To create a consumer group

Pub/Sub

• Publisher: A client sends messages to a specific “channel”.
• Subscriber: Clients subscribe to one or more channels to receive messages.
• No Persistence: Redis Pub/Sub does not store messages. Once a message is sent, it is delivered to all subscribers in real time, but it’s not saved or re-delivered if a subscriber is not connected.
• No ack from client there is chance of data loss

 
PUBLISH mychannel "Hello, World!"
 
SUBSCRIBE mychannel
 

What It Means to “Store Data in Memory”

When a program runs, it gets a chunk of RAM (volatile memory). Inside that memory, all data — numbers, strings, lists — are stored as bytes. Each piece of data has a memory address, like a street address, telling the CPU where it lives.

In a language like C (which Redis is written in):

• You don’t have “objects” like in Python or JavaScript.
• You have raw memory blocks (pointers to bytes).
• You manually manage them with malloc (allocate) and free (release).

So if we want to store:

key = "user:1"
value = "Alice"

Redis must allocate two chunks of memory (for key and value), and keep some structure that remembers:

• where each key lives,
• what type of value it has,
• and how to find it quickly.

That’s where the hash table comes in.

The Core Structure Hash Table

Redis’s entire dataset lives inside a dictionary (called dict in the source code). A dictionary is just a hash table — a data structure mapping keys to values efficiently.

Mechanism:

1. Take the key string.
2. Compute a hash value (a number derived from the string).
3. Use that hash to find the bucket (array index) where the key-value pair is stored.

Example:

Hash("user:1") → 17
Store at dict->table[17]

Inside each bucket, Redis stores a linked list (chain) of entries that hash to the same slot (to handle collisions).

Each entry (called dictEntry) looks like:

struct dictEntry {
    void *key;        // pointer to key string
    void *val;        // pointer to value object
    dictEntry *next;  // pointer to next entry in case of hash collision
};

So Redis’s main in-memory structure is:

RedisDB
 └── dict (hash table)
      ├── table[] array of pointers (buckets)
      └── linked lists of key-value entries

Memory Allocation

Redis uses jemalloc (a fast general-purpose allocator) instead of the system malloc, because:

• It reduces fragmentation.
• It groups small objects together efficiently.
• It reuses freed blocks faster.

Whenever a new key or value is inserted, Redis calls malloc() to reserve memory. All structures — strings, lists, hash tables, etc. — are just different shapes of memory layouts.

The Redis Object System (robj)

Every key and value in Redis isn’t stored as raw bytes — they’re wrapped in a structure called a Redis object (robj). This gives Redis flexibility to handle different data types.

struct redisObject {
    unsigned type:4;      // string, list, hash, set, etc.
    unsigned encoding:4;  // how it’s stored in memory
    void *ptr;            // pointer to actual data
    int refcount;         // for memory management
};

So the val pointer in dictEntry doesn’t directly point to data — it points to a robj, which in turn points to the real data.

Example:

key = "user:1"
value = "Alice"

dictEntry
 ├── key → robj(type=string, ptr="user:1")
 └── val → robj(type=string, ptr="Alice")

This layer of indirection allows Redis to switch encodings internally (e.g., store short strings compactly, large ones differently) without changing higher-level logic.

Different Encodings for Different Data Types

Depending on how you use a key, Redis chooses the most memory-efficient encoding.

Data Type	Common Encodings	Description
String	int, embstr, raw	Stores integers directly; short strings inline; long strings as separate memory blocks
List	quicklist	Combination of linked list + small arrays
Set	intset, hashtable	Small sets as compact arrays; large sets as hash tables
Hash	ziplist, hashtable	Small hash as flat array of key–value pairs; large hash as full dict
Sorted Set	ziplist, skiplist+dict	Compact or full dual structure (for ordering + fast lookup)

Redis dynamically upgrades encodings:

• A small list starts as compact bytes (ziplist).
• When it grows, Redis automatically converts it to a full linked structure (quicklist).

This adaptability is key to Redis’s memory efficiency.

Rehashing (Dynamic Resizing)

As the number of keys grows, Redis resizes its hash table to keep lookups fast (O(1)).

Instead of stopping the world to rebuild the table, Redis does incremental rehashing:

• It keeps two tables (ht[0] and ht[1]).
• Gradually moves entries from the old to the new one as commands run.
• Ensures smooth scaling without latency spikes.

Persistence (Saving to Disk)

Redis is in-memory but supports persistence for recovery.

Two mechanisms:

1. RDB (snapshot): periodically dumps the whole dataset to disk.
2. AOF (Append Only File): logs every write operation for replay later.

Internally, this is independent of how data is stored in memory. Redis serializes each in-memory object and writes it out in a compact format.

Access Path (How a GET Works)

Let’s trace a command:

GET user:1

Step-by-step:

1. Redis parses the command.
2. Computes hash of “user:1”.
3. Finds corresponding bucket in the main hash table.
4. Traverses linked list (if collision) to find matching key.
5. Returns the value’s robj->ptr.

All in memory no disk, no locks — so this entire path runs in microseconds.

Mental Model Summary

Think of Redis as a tiny in-memory operating system managing different containers of data, all built on the same foundation — hash tables and encoded memory layouts.

RedisDB
 ├── Main dict: { key → redisObject(value) }
 ├── Each redisObject has (type, encoding, ptr)
 ├── Each ptr points to structure (string / list / set / hash)
 └── Each structure has optimized internal representation

Data types

Redis keys

• Keys: Keys are unique, binary-safe, and often use structured naming conventions (e.g., event:judo) for better organization
• Binary safe mean when comparing they use binary format to compare
• Keys can also be set to expire automatically, useful for temporary data like session stores

Covention for creating keys

• we can use colons (:) or other separators to create a hierarchy within our key names. For example, domainObject:instance:attribute.

• The speaker gives examples like event:Judo and venue:Stirling 

• Sometimes, plurals are used for lists or collections

• It’s crucial to have a consistent naming convention and stick to it to avoid clashes, especially when different services co-locate data in the same Redis instance

• Remember that Redis key comparisons are binary safe, meaning fencing, Fencing, and FENCING are all considered different keys

• Strings: Can store various data types, which Redis internally optimizes

• Lists: Doubly-linked lists supporting operations from the left (head) or right (tail), efficient for queues and stacks

• Sets: Unordered collections of unique values, useful for tag clouds or unique visitor counts, and support set operations like union and intersection

• Sorted Sets: Ordered collections of unique strings, where elements are ordered by a numerical score, ideal for leaderboards or priority queues

• Hashes: Key-value stores within a key, providing a single level of hierarchy for values, good for dynamic data feeds or session caches

LIST

• Structure: Redis lists are implemented as doubly linked lists 

• This means you can traverse them efficiently in both directions (forward and backward).

• Naming Convention: Unlike traditional “head” and “tail” for lists, Redis uses “left” and “right” to refer to the ends of the list . You’ll see this reflected in command names.

• Key Operations:

  • LPOP: Removes and returns an element from the left side (start) of the list
  • RPOP: Removes and returns an element from the right side (end) of the list
  • commands like LPUSH and RPUSH exist to add elements to the left or right, respectively

• Use Cases: Lists are incredibly versatile and can be used to create:

  • Queues: By pushing elements to one end (e.g., right with RPUSH) and popping from the other (e.g., left with LPOP).
  • Stacks: By pushing and popping from the same end (e.g., left with LPUSH and LPOP).
  • Capped Lists: A variant of lists where you can ensure only a certain number of elements are retained . This is great for keeping a limited number of recent items, like the last 5 posts in an activity stream

• Performance (Time Complexity): A major advantage of Redis lists is their performance, specifically their O(1) time complexity for operations like adding or removing elements from either end .

  • O(1) means the time it takes to perform the operation is constant, regardless of how large the list is Whether the list has 3 elements or a billion, getting or adding the first/last element takes roughly the same fixed time.
  • This consistent performance makes them ideal for inter-process communication and activity streams

Sets

• Unordered and Unique: A Redis Set is an unordered collection of unique strings. This means that each element in a set can appear only once, and the order in which you add elements does not matter when you retrieve them.

• Key Operations:

  • Adding elements: You can add one or more elements to a set. If you try to add an element that’s already in the set, it’s ignored, maintaining uniqueness.
  • Retrieving members: You can fetch all members of a set using a command like SMEMBERS.
  • Checking membership: You can quickly check if a specific element is already a member of a set.
  • Removing elements: You can remove specific elements from a set.

• Set Operations: One of the most powerful features of Redis Sets is their ability to perform standard set theory operations between multiple sets:

  • Union: Combines all unique elements from two or more sets.
  • Intersection: Returns only the elements that are common to all specified sets.
  • Difference: Returns elements present in the first set but not in the subsequent sets.

• Typical Use Cases:

  • Tagging: Keeping track of unique tags associated with objects.
  • Unique Visitors: Easily count unique visitors to a website during a specific period by adding their unique IDs (like cookie IDs) to a set.
  • User Permissions: Managing unique roles or permissions for users.
  • Finding Commonalities/Differences: Using union, intersection, and difference operations to analyze relationships between different groups of data, such as a taxonomy of tags.

• Performance: Many of the operations on Redis Sets, including adding, removing, and checking for membership, have O(1) time complexity. This means their performance is very consistent and fast, regardless of the size of the set.

Ordered sets

Redis Sorted Sets (ZSETs) are similar to regular Sets in that they are collections of unique strings. However, their defining feature is that each member is associated with a numerical score, and this score is used to order the members.

• Ordered Collection of Unique Strings: Like regular Sets, Sorted Sets only store unique members. The key difference is that they maintain an order based on a score.

• Scores for Ordering: When you add a value (member) to a sorted set, you must also specify a floating-point number as its score. This score dictates the member’s position within the set.

• Automatic Reordering: If a member’s score changes (e.g., a player’s points in a game are updated), the sorted set automatically reorders itself to reflect the new position of that member.

• Retrieval Flexibility: You can retrieve members from a sorted set in their correct order, either from highest score to lowest or lowest score to highest. You can also retrieve members by position, value, or within a specific score range.

• Set Operations: Sorted sets also support set operations like Union and Intersection, similar to regular sets, allowing you to combine or compare ordered collections.

• Typical Use Cases:

  • Leaderboards: This is a classic example. The member could be a player’s screen name, and the score would be their total points. As scores update, the leaderboard stays correctly ordered.
  • Priority Queues: You can use the score to represent the priority of items in a queue. When you pop elements off the queue, you can get them in priority order.
  • Ranking Systems: Any scenario where you need unique items ranked by a certain metric.

• Performance: Operations on Sorted Sets typically have a time complexity of O(log N), which means the time taken grows logarithmically with the number of elements (N). This is still very efficient, providing consistent performance even with large datasets.

Hashes

Redis Hashes are essentially like a mini key-value store encapsulated within a single Redis key. You can think of them as an object or a dictionary.

Here’s how they work:

• Structure: Each Redis Hash key maps to a collection of field-value pairs. So, a single Redis key can hold multiple distinct fields, each with its own associated value.

  • For example, you might have a Redis key named event:judo, and within that key, you store fields like venue (with a value of “Superdome”), capacity (with a value of “32000”), and subway_line (with a value of “Yes”).

• Dynamic and Schema-less: You don’t need to define the fields upfront. You can add and remove fields dynamically within a hash at any time. This flexibility is particularly useful if you’re dealing with data feeds or IoT devices where the attributes you receive might change or vary.

• Key Operations:

  • HSET: Sets the value of a field in a hash. If the field doesn’t exist, it’s created.
  • HGET: Retrieves the value associated with a specific field from a hash.
  • HGETALL: Retrieves all field-value pairs from a hash.
  • HDEL: Deletes one or more fields from a hash.
  • HINCRBY: Increments the integer value of a field by a given amount.

HSET call:123 \
    company_id 77 \
    state RINGING \
    assigned_agent null \
    created_at 17000000

//Read a single field:
HGET call:123 state

• Typical Use Cases:

  • Storing Objects: They are a great way to represent objects where each attribute of the object is a field within the hash.
  • Data Feeds: Effectively consume and store data from feeds where the structure or attributes might change frequently.
  • IoT Device Attributes: Store specific attributes reported by an IoT device.
  • Rate Limiting: You can use fields within a hash as counters (e.g., for API endpoints) to track usage and implement rate limits.
  • Session Caches: Store session-related information, normalizing individual fields into the hash rather than storing a large binary string.

• Performance: Many operations on Redis Hashes, such as getting or setting individual fields, have O(1) time complexity. This means that accessing a field in a hash with one field is just as fast as accessing a field in a hash with a million fields. This provides very consistent and predictable performance.

LOCKS

Lock Type	What It Is	How It Works	Why/When to Use	Example (Pseudo-code/Usage)
1. Simple Lock	Basic mutual exclusion on a single Redis instance using one key.	Use SET key value NX EX TTL so only one client can set; others fail. TTL prevents permanent lock.	Fast, lightweight coordination for short non-critical sections. Not safe under failover.	SET "order:123" uuid NX EX 30 → do work → DEL "order:123"
2. Safe Simple Lock	Simple lock plus ownership verification.	Same as Simple Lock, but store a unique token; only delete if token matches (usually via Lua).	Prevents accidental unlock by non-owners; safer than plain Simple Lock.	Acquire: SET lock uuid NX EX; Release: Lua script checks token before DEL
3. Redlock (Distributed Lock)	Lock across multiple independent Redis nodes for higher safety.	Try to set the lock on N nodes; must get majority and completion within TTL.	Resilient to node failure; no single point of failure. Better than single-node but not perfect.	Acquire on 5 nodes, require at least 3 successes within TTL.
4. Reentrant Lock	Same client can re-acquire the same lock without deadlocking itself.	Store owner + counter. If the same owner re-enters, increment counter. Only free when counter hits 0.	Needed when code routines call each other and both require the same lock.	Hash with fields: owner, count → increment if same client; decrement on release.
5. Read-Write Lock	Allows many readers, but only one writer at a time.	Track read count + writer state; readers allowed only if no writer, writer only if no readers.	Great for read-heavy workloads where reads don’t conflict.	Reader: check no writer, then increment count; Writer: wait until count=0 and no writer.
6. Semaphore Lock	Limit N concurrent holders (not exclusive).	Use sorted sets: add client with timestamp, trim expired ones, check rank < limit.	When you want controlled concurrency, like limited threads or API quotas.	ZADD sem_key now client_id → Check ZRANK < limit → proceed or retry.
7. Fair Lock (FIFO)	Ensures first come, first served order of acquisition.	Use a queue (streams/lists); clients wait for head position before locking.	When fairness and no starvation matter (ticketing, priority systems).	XADD queue * client_id → wait until you’re head → acquire lock.
8. Fencing Token Lock	Prevent “zombie” clients from corrupting state after expiration.	Each lock returns a monotonically increasing token. Resource rejects operations with old tokens.	Critical systems where stale locks must never corrupt data.	Acquire → get fencing token; resource compares token > last seen.
9. Sharded / Multi-Key Lock (Striped Lock)	Split a big lock into many smaller locks to reduce contention.	Hash resource to a stripe → lock that stripe.	High-concurrency systems with many distinct resources.	stripe = hash(key)%256 → lock lock:stripe:42.
10. Multi-Resource (Composite) Lock	Lock multiple keys in one operation without deadlocks.	Determine deterministic order → acquire locks in that order → release.	Transactions requiring multiple resource locks.	Sort resource IDs → acquire each lock in order → work → release.
11. Auto-Renewal Lock	Extend TTL periodically so long jobs don’t expire their lock.	Background loop calls EXPIRE to renew lock, checking ownership.	Long-running jobs where static TTL isn’t enough.	Acquire lock → spawn thread to EXPIRE periodically → release at end.
12. Leased Lock	Time-bound lease with explicit renewal by client.	Lease returns token + expiry; client must renew before TTL if needed.	Predictable time-bounded access, safe against runaway processes.	Request lease with specific duration → use within TTL → optionally renew.

Locks edge case handled with lua

Imagine you write this logic:

1. Check if I own the lock
2. If yes → delete it

if GET lock_key == my_id
   DEL lock_key

But now imagine two processes:

• Process A (you)
• Process B (another service)

And the timeline:

1. A reads GET → sees it owns lock
2. Lock expires
3. B acquires lock
4. A runs DEL

Now A just deleted B’s lock.

Redis guarantees:

Each individual command is atomic.

But it does NOT guarantee:

A sequence of commands is atomic.

We need:

Check and delete must happen as one indivisible operation.

Meaning:

• Either both happen
• Or neither happens
• And nothing can interrupt between them.

So redis support executing the lua script when we pass the script it excute as single unit

if redis.call("GET", key) == value then
  return redis.call("DEL", key)
end

Redis executes the whole script:

• As a single unit
• Without interleaving other commands
• On the same thread

Transcation

Transaction Type	What It Is	How It Works	Why / When to Use	Example
1. MULTI / EXEC (Basic Transaction)	Groups multiple commands to execute atomically.	MULTI starts queue → commands queued → EXEC executes all sequentially without interruption.	When you need atomic batch updates.	MULTI INCR balance ECR inventory EXEC
2. DISCARD	Cancels a queued transaction.	After MULTI, if needed, call DISCARD to clear queue.	Abort before execution if condition changes.	MULTI SET x 10 DISCARD
3. WATCH (Optimistic Locking)	Conditional transaction execution based on key changes.	WATCH key → if key changes before EXEC, transaction aborts.	Prevent race conditions without locking.	WATCH balance GET balance MULTI SET balance 90 EXEC
4. UNWATCH	Removes watched keys.	Clears watch state manually.	Stop optimistic locking before EXEC.	UNWATCH
5. Lua Script Transaction	Atomic logic using embedded scripting.	EVAL runs entire Lua script atomically.	When you need logic + atomicity together.	EVAL "redis.call('INCR','x')" 0
6. Pipeline (NOT a transaction)	Batches commands to reduce round trips.	Sends multiple commands without waiting for replies.	Performance optimization only (not atomic).	Client sends 100 commands at once.
7. CAS Pattern (WATCH + MULTI)	Compare-and-set pattern using WATCH.	Read → modify → EXEC fails if value changed.	Bank balance, counters, conditional updates.	Classic optimistic concurrency control.
8. Distributed Transaction (App-level)	Coordinating multiple Redis instances manually.	App logic ensures consistency.	Multi-node workflows.	Use Redlock + MULTI pattern.

Why WATCH Exists Redis does NOT isolate reads.

So this happens:

Client A: GET x → 10
Client B: SET x 20
Client A: MULTI
Client A: SET x 11
EXEC

Without WATCH → A overwrites B.

WATCH solves this by:

• Marking keys
• If any watched key changes before EXEC → transaction aborts

This is optimistic concurrency control.

Reaminder

ZADD reminders 1710000000 reminder:123
SET reminder:123 {...}

Resources

• https://mattermost.com/blog/lessons-learned-running-redis-at-scale/
• Redis Data Structures for Non-Redis Users