Caching strategy, I tried to understand
Situation/context:
I got to know about a situation called cache stampede. In this scenario, a lot of requests when misses the cache and all of them hits the DB to get the data. I got to know about polling locks and stale while revalidate to mitigate this issue. How we might scale it for higher number of users. Now, all of these have not been implemented by me yet, but I wanted to know, if some situation comes, how I can handle.
scenario:
We got 200 simultaneous requests at the same time.
polling locks:
In this solution, whenever a cache miss happens, we lock the redis with an expiry time of say 10s. We need this time to low so that we cannot go under deadlock, otherwise if lock persists for more time then we might have some other issues like the other 199 requests might be in a longer polling session, which might increase memory and CPU usage.
The other 199 requests keep on polling until the locked request updates the cache key with fresh data.
I am thinking using FastAPI. Following might be the steps:
- 1st request arrives
- Checks the redis
- if available
- returns the cached data
- if not (Step #3)
- creates a lock context using “async with” with a lock key and expiry of 10s
- acquires lock (Step #3.2)
- fetches from DB
- rebuilds the cache with new data
- returns the new data and releases the lock
- while the 1st request acquires the lock, there comes 3 requests simultaneously
- these requests will fail to acquire lock at “Step #3.2”
- lock acquire fails (Step #3.3)
- the request will be put to sleep for “0.1s”
- after each sleep completion, it will keep checking the redis, if cache is built by request #1 or not
- if its a hit
- returns the cached data
- else
- it will continue to sleep again for 0.1s
- this might continue for 5s
- if within 5s, it still not get the data, it will return error
- it will continue to sleep again for 0.1s
- if for some reason, request #1 fails within the 5s of polling, request #2 will acquire lock and try to build the cache from the DB
- as we have 3 simultaneous requests after request #1, it might happen all of them try to get the lock again simultaneously
- to prevent this, we generally use jitter, a small random number difference for each sleep of the polling rather than the 0.1s
- await asyncio.sleep(random.uniform(0.05, 0.15))
- lock acquire fails (Step #3.3)
app.get("/data")async def get_data(redis: aioredis.Redis = Depends(get_redis)): cache_key = "expensive_data" lock_key = f"{cache_key}:lock" # 1. Check cache first cached_value = await redis.get(cache_key) if cached_value: return {"source": "cache", "data": cached_value} # 2. Try to acquire the lock (Max life 10 seconds to prevent deadlocks) # blocking=False ensures we don't block the actual Python event loop thread async with redis.lock(lock_key, timeout=10, blocking=False) as lock: if await lock.acquire(): try: # Double-check cache inside the lock (in case a waiting request just finished it) cached_value = await redis.get(cache_key) if cached_value: return {"source": "cache_after_wait", "data": cached_value} # Rebuild the cache db_data = await fetch_data_from_db() await redis.set(cache_key, db_data, ex=60) # Cache for 60 seconds return {"source": "db_build", "data": db_data} finally: await lock.release() else: # 3. If lock is held by another request, poll until the data is ready # This is where subsequent "stampede" requests wait safely for _ in range(50): # Retry for up to 5 seconds (50 * 0.1s) await asyncio.sleep(0.1) cached_value = await redis.get(cache_key) if cached_value: return {"source": "cache_after_wait", "data": cached_value} # Fallback if the holding worker crashed/timed out completely return {"error": "Timeout waiting for cache to regenerate"}
Drawbacks:
This works fine for small to medium users, but it will create issue, when users increase. As I discussed earlier,
- memory size might increase, if a lot of requests keeps polling as we are using loop. We are actually keeping all those requests in our memeory alive and looping them one by one, making them sleep and waking up.
- Eventually, connection limit light also increase as the user requests grows
stale while revalidate:
In this solution, we will not be keeping the requests in our memory, but we will respond them with the stale data while, we build the cache in the background.
Following might be the steps:
async def fetch_and_cache_data(redis: aioredis.Redis, cache_key: str): """Background task that fetches fresh data and updates Redis.""" try: # 1. Fresh Data from DB fresh_data = await fetch_from_db() # 2. Package data with the current timestamp payload = { "data": fresh_data, "generated_at": time.time() } # 3. Save to Redis with a long hard TTL (e.g., 1 day) so it never completely vanishes await redis.set(cache_key, json.dumps(payload), ex=86400) print(f"Cache {cache_key} successfully revalidated!") except: # Clear the background lock so future runs can execute await redis.delete(f"{cache_key}:bg_lock")app.get("/data")async def get_data(background_tasks: BackgroundTasks, redis: aioredis.Redis = Depends(get_redis)): cache_key = "expensive_data" soft_ttl = 60 # We want data to refresh every 60 seconds # 1. Look up the cached payload raw_cached = await redis.get(cache_key) # CASE 1: COLD START (Cache is completely empty) # Fallback for the absolute first time the app ever runs (Cold Start) if not raw_cached: # Try to acquire the lock to be the ONE worker building it lock_acquired = await redis.set(f"{cache_key}:bg_lock", "processing", ex=15, nx=True) if lock_acquired: background_tasks.add_task(fetch_and_cache_data, redis, cache_key) # ❌ DON'T HANG! Instantly send 202 Accepted and tell frontend to poll return { "status": "processing", "message": "Cache is building for the first time.", "retry_after_seconds": 2 } # CASE 2: CACHE EXISTS (Calculate the Soft TTL Time Difference) cached_payload = json.loads(raw_cached) age = time.time() - cached_payload["generated_at"] # 2. Check if the data is "stale" (older than our soft TTL) if age > soft_ttl: # Use a secondary Redis lock key with a short TTL (e.g., 10s) # to ensure ONLY ONE background worker is triggered lock_acquired = await redis.set(f"{cache_key}:bg_lock", "locked", ex=10, nx=True) if lock_acquired: print("Cache is stale! Triggering background revalidation...") # Hand off the heavy DB work to FastAPI's background thread pool background_tasks.add_task(fetch_and_cache_data, redis, cache_key) # Note: We do NOT return 'processing' here. # Because we have stale data available, we instantly give it to the user. return { "status": "stale", "data": cached_payload["data"], "message": "Here is old data, updating in background...", "retry_after_seconds": 2 } # CASE 3: HAPPY PATH (Data is under 60 seconds old) # 3. INSTANT RETURN: Both Group 1 (Fresh) and Group 2 (Stale) users exit here immediately! return {"status": "fresh", "data": cached_payload["data"], "age_seconds": int(age)}
In a nutshell,
- keep the data from the backend in a custom dict with db data and current time
- convert this dict to a string and store in the redis cache with a large ttl of around 1 day
- when requests come
- set a small time that will act as ttl logicaly rather than applying it in redis, for example 60s
- Scenario 1: Single request arrives
- get the cached data
- if cache miss
- acquire redis lock
- push the cache hydrate function to the background task
- return immediately with status as processing and message saying to try after certain time
- if cache hit
- check if the data is stale or not by comparing the diff between current time and stored time in the cache data with soft ttl
- if stale
- acquire lock
- push the cache hydrate function to the background task
- return status with stale and retry after certain time
- if fresh
- return the data with status as fresh
- Scenario 2: Multiple requests hit simultaneosuly at the same time
- the same thing happens as it happened for scenario 1 for the first request
- first request will lock redis, start background task and return immediately with status as processing
- all other will also return status with processing as we they cannot acquire lock due to “nx=True”
The frontend need to handle these status accordingly and keep on polling till it gets the status as “fresh” for the cold start when the cache is built up for the first time
Except the cold start, frontend need not to be polled, but can be initimated via SSE or websockets, saying that background task is completed and fresh data can be refetched once background task is completed. Thats what modern apps does, when fresh data is cached, a popup saying “new posts” or “refresh” is displayed in frontend.
When million users requests, cold start is a real issue, which might be mitigated by introducing CDNs like cloudflare or API gateway like nginx before fastapi server.
After 1st request of a cold start, redis lock and background task, it returns cache control and max age = 5 in headers to the CDN or gateway, which will absorb the (million – 1) requests by saying them to wait for 5 secs
We can also pre warm the cache before app starts so that the common data are already present in cache before user start requesting it. This should be done separately from the app’s code.
Other than the cold start, I think, we wont be having a cache issue as the stale data will be delivered to the user.
For such large user base, we might move to celery as fastapi’s built in background task might not be able to handle them.
Mahesh 
Leave a comment