Caching: The First Tool of System Design

When a system slows down under load, the instinct is to scale the database. The cheaper move is almost always to stop asking the database the same question a thousand times a second. Caching is the first tool of system design because read traffic dominates most workloads — often by a factor of a hundred.

The four places a cache can live

1. Browser and CDN: static assets and public pages. The best request is one that never reaches you.
2. Application memory: per-process caches for configuration and hot lookups. Fast, free, but inconsistent across instances.
3. Distributed cache (Redis, Memcached): shared across instances, the workhorse for session data, computed results, and hot rows.
4. Database internals: materialized views and query caches — useful, but the least under your control.

Cache-aside: the pattern to learn first

The application checks the cache, falls back to the database on a miss, and writes the result back with a TTL. It is simple, tolerates cache failure gracefully, and covers the majority of use cases.

async def get_profile(user_id: int) -> Profile:
    key = f"profile:{user_id}"
    if cached := await redis.get(key):
        return Profile.model_validate_json(cached)

    profile = await db.fetch_profile(user_id)
    # TTL bounds staleness; jitter prevents synchronized expiry stampedes.
    ttl = 300 + random.randint(0, 60)
    await redis.set(key, profile.model_dump_json(), ex=ttl)
    return profile

Invalidation: choose your staleness budget

The famous hard problem is really one question: how stale is acceptable? Answer it per data type, explicitly. Product descriptions can be five minutes stale; account balances cannot. TTL alone covers the first case. For the second, delete the cache key inside the same transaction boundary as the write — and accept that you now own a consistency edge case during failures.

The failure modes interviewers ask about

• Stampede: a hot key expires and a thousand requests hit the database at once. Fix with TTL jitter or a single-flight lock.
• Penetration: requests for keys that never exist bypass the cache entirely. Fix by caching the negative result briefly.
• Hot key: one celebrity row overwhelms a single cache node. Fix by replicating that key locally in application memory.

Every cache is a bet that the past predicts the near future. Know exactly how much you are betting, and what losing costs.