# When Cache Expiry Crashes Your System: The Thundering Herd Problem Imagine hundreds of people waiting outside a store for it to open during a flash sale. The moment the doors open, everyone rushes inside at the same time, overwhelming the staff and slowing down service for everyone. A similar situation occurs in distributed systems when many clients attempt to access the same resource simultaneously. This phenomenon is known as the **Thundering Herd Problem**, and it can cause sudden spikes in load that overwhelm backend systems. ## What is the Thundering Herd Problem? The **Thundering Herd Problem** occurs when a large number of clients or processes attempt to perform the **same operation at the same time**, overwhelming a shared system resource. This usually happens when many clients are waiting for the **same event**. When that event occurs, all waiting clients wake up simultaneously and try to access the resource at once. Instead of requests being **spread over time**, they become **synchronized**, causing a sudden spike in load. This problem commonly appears in distributed systems in situations such as: * **Caching systems** – when a popular cache entry expires and many requests try to rebuild it at once * **Databases** – when multiple services issue the same queries simultaneously * **Load balancers** – when many queued requests are routed to a server that just became available * **Distributed locks** – when multiple processes attempt to acquire a lock immediately after it is released Because many requests target the **same backend resource simultaneously**, the system can experience increased latency, high CPU usage, and database overload. ## How Requests Flow Through Cache and Database Most modern applications use a **cache layer between the application and the database** to reduce database load and improve response times. When a client sends a request, the application first checks the cache before querying the database. ### Cache Hit Flow ![Cache Hit Flow](https://cdn.hashnode.com/uploads/covers/62c50f195527bf5d12a26bf7/92f7f381-f8a3-410d-8caa-0c196b55075c.png align="center") When the requested data is already present in the cache, the application can immediately return the cached data to the client without querying the database. ### Cache Miss Flow ![](https://cdn.hashnode.com/uploads/covers/62c50f195527bf5d12a26bf7/f46e7521-b22b-4a15-92f4-090fe8c2ae24.png align="center") If the data is not present in the cache, the application queries the database, retrieves the result, and stores it in the cache for future requests. ## What Happens When the Cache TTL Expires In the previous diagram, we saw how a typical **cache-aside flow** works. The application first checks the cache for the requested data. If the data is present (**cache hit**), it is returned immediately without querying the database. If the data is not present (**cache miss**), the application queries the database, stores the result in the cache, and then returns the response to the client. Caches usually store data with a **TTL (Time To Live)**. TTL determines how long a cached entry remains valid before it expires. When the TTL expires, the cached data is removed from the cache. The next incoming request will therefore experience a **cache miss**, triggering the flow shown in the diagram: 1. The client sends a request to the application. 2. The application performs a **cache lookup**. 3. Because the cached entry has expired, the lookup results in a **cache miss**. 4. The application queries the **database** for the required data. 5. The database returns the result to the application. 6. The application **populates the cache** with the new data. 7. The response is returned to the client. In normal situations, this process works well because cache misses occur **occasionally and independently**. Only a small number of requests need to query the database before the cache is repopulated. However, problems arise when **many requests arrive at the same time immediately after the cache entry expires**. In that situation, multiple requests may simultaneously experience a cache miss and all attempt to query the database. This is where the **Thundering Herd Problem** begins to appear. ## How Cache Expiry Leads to a Thundering Herd The cache flow described earlier works well when requests arrive gradually. When a cache entry expires, only a small number of requests experience a cache miss, query the database, and repopulate the cache. However, in high-traffic systems, a different situation can occur. Imagine a cache entry that is frequently requested by many users — for example: * a popular product page * trending posts on a social platform * a frequently accessed configuration value Suppose this cache entry has a **TTL of 5 minutes**. When the TTL expires, the cached value is removed. Now consider what happens if **many clients send requests at the same moment after the cache expires**. All of those requests follow the same flow shown in the previous diagram: 1. Each client sends a request to the application. 2. The application performs a **cache lookup**. 3. Since the cache entry has expired, every request experiences a **cache miss**. 4. Each request independently queries the **database**. 5. The database now receives a **large number of identical queries simultaneously**. Instead of a single request repopulating the cache, the database suddenly receives **hundreds or thousands of identical queries at the same time**. This surge of synchronized requests is what creates the **Thundering Herd Problem**. The result is a sudden spike in load on the backend system. The database must process many redundant queries, application servers consume additional CPU resources, and response latency increases for all users. In severe cases, this burst of simultaneous requests can overwhelm the database and lead to cascading failures across the system. ## Impact of the Thundering Herd Problem When a thundering herd occurs, many requests attempt to perform the same expensive operation at the same time. Instead of spreading the load across time, the system experiences a sudden burst of identical work. This can have several effects on different parts of the system. ### CPU Usage Application servers must process a large number of requests simultaneously. If many of these requests trigger database queries or complex computations, CPU utilization can spike rapidly. ### Database Load The database is often the most heavily affected component. When many requests bypass the cache simultaneously, the database receives a large number of identical queries at once. This can exhaust connection pools and significantly increase query latency. ### Cache Effectiveness During a thundering herd event, the cache temporarily loses its effectiveness. Since many requests encounter a cache miss at the same time, the system behaves as if the cache layer does not exist. ### Increased Latency As backend resources become overloaded, requests begin to queue up. This increases response times for users and can lead to timeouts or failed requests. In extreme cases, the sudden load spike can cause **cascading failures**, where one overloaded service slows down other dependent services in the system. ## Techniques to Prevent the Thundering Herd Problems Because the thundering herd problem is common in distributed systems, several techniques are used to reduce or prevent it. ### Request Coalescing Instead of allowing multiple identical requests to query the database, the system combines them into a single backend request. Other requests wait for the result and reuse the same response. ### Cache Locking (Mutex) A locking mechanism ensures that only one request is allowed to regenerate a missing cache entry. While that request fetches the data from the database, other requests wait until the cache is repopulated. ### Staggered Cache Expiry If many cache entries expire at the same time, synchronized cache misses can occur. Adding small random variations to cache expiration times helps distribute cache refresh operations across time. ### Exponential Backoff When clients retry failed requests, they should wait for increasing time intervals between retries. This prevents large numbers of clients from retrying simultaneously. ### Rate Limiting Rate limiting controls how many requests can reach backend services within a certain period of time. This protects critical components like databases from sudden bursts of traffic. ## Conclusion The **Thundering Herd Problem** occurs when many clients attempt to perform the same operation simultaneously, often triggered by events such as cache expiry or lock release. When this happens, the sudden burst of identical requests can overwhelm backend systems like databases and application servers. Understanding how this behavior emerges—especially in caching architectures—is important when designing scalable distributed systems. Without proper safeguards, even a small synchronization event can lead to large spikes in load and degraded system performance. Techniques such as **request coalescing, cache locking, staggered cache expiry, and rate limiting** help prevent these synchronized bursts of work and keep systems stable under heavy traffic. By anticipating these scenarios and designing systems that distribute load over time, engineers can ensure that their services remain **reliable, responsive, and scalable** even under high demand.