Designing a Scalable URL Shortening Service

URL shorteners look deceptively simple.

Given a destination such as:

https://example.com/products/2026/catalog?category=distributed-systems&region=europe

the service produces a compact address:

https://sho.rt/Ab91xK2

Opening that address retrieves a stored mapping and returns an HTTP redirect. A single application and database can perform both operations.

The interesting design questions appear when the link has a lifecycle. A customer wants a recognizable alias, a campaign must stop redirecting at midnight, an administrator must block a malicious destination, and a creator must be able to retry a timed-out request without producing another link. Meanwhile, a popular URL can receive more traffic than the rest of the service combined.

These requirements connect the write and read paths. Identifier allocation determines how records can be partitioned. Expiration and blocking determine what a cache may serve. Event-delivery guarantees determine whether analytics can remain independent of redirects.

This article develops that design from requirements through capacity planning, APIs, storage and failure handling, using a workload that can accumulate hundreds of billions of mappings.

Table of Contents

• 1. Requirements
• 2. Capacity Estimation
• 3. API Design
• 4. High-Level Architecture
• 5. Short Code Generation Strategy
• 6. Data Model, Write Path & Consistency
• 7. Read Path & Caching
• 8. Scaling, Reliability & Security
• 9. Analytics, Evolution & Trade-offs
• Conclusion

1. Requirements

Functional requirements

The service supports four operations:

1. Create a link. Accept a destination URL, an optional custom alias and an optional expiration timestamp. Generate a short code when no alias is supplied.
2. Resolve a link. Redirect only while the link is active and unexpired, whether its public address contains a generated code or a custom alias.
3. Block a link. Allow an authorized administrator to disable resolution, with a documented maximum propagation delay.
4. Collect usage statistics. Record resolution events asynchronously, accepting bounded event loss for approximate analytics.

A link can use either kind of public name, with or without an expiration date. Every combination follows the same ownership, blocking and retry rules.

Destinations, public names and expiration timestamps are immutable after creation. Administrative status can change. We do not offer user deletion or destination editing, and we never reassign an issued public name. Multiple links may point to the same URL because their ownership, expiration and analytics can differ.

Creation uses an authenticated account or API key; identity-provider implementation, billing and the analytics dashboard are outside this design. Redirects are public. Knowing a short URL is not an authorization mechanism for confidential resources.

Non-functional requirements

The availability target corresponds to about 4.4 hours of downtime per 365-day year. Client network time and destination-page loading are outside the latency budget. Expected responses for unknown, expired or blocked links are distinct from infrastructure failures when calculating service availability.

A takedown freshness budget must also be chosen before deployment. It includes replication delay, cache lifetime and invalidation propagation. The architecture must enforce that bound even when an invalidation message is missed.

2. Capacity Estimation

Suppose the service creates approximately 100 million short URLs per day.

That gives us:

100,000,000 / 86,400 ≈ 1,157 writes/second

Rounding up, the write workload is roughly:

~1.2K writes/sec

URL resolution is generally much more frequent than URL creation. If we assume a 10:1 read-to-write ratio:

1,157 × 10 ≈ 11,570 reads/sec

So our baseline workload is approximately:

Average traffic isn’t the whole story. Production systems experience bursts, so the architecture should have enough headroom to absorb traffic several times higher than these averages.

Traffic distribution and peak load

Our capacity model starts from 100 million new mappings per day over ten years. Creation volume, retained mappings and active users measure different things. Keep those quantities separate so that traffic and storage estimates describe the same workload.

Daily active users are not requests per second: converting one into the other requires an assumed number of resolutions per user per day. Likewise, a peak multiplier is a scenario to test, not a consequence of the daily average. Test the sensitivity to different access patterns; for example, 100:1 at the same creation rate implies about 115.7K average resolutions per second.

Because this design permits several links for one destination, the number of distinct destination URLs does not bound the number of mapping records.

Storage growth

Assume the service operates for ten years:

100M × 365 × 10
≈ 365 billion URLs

If the average original URL consumes roughly 100 bytes of payload storage:

365B × 100 bytes
≈ 36.5 TB

This is only an order-of-magnitude estimate. A real capacity model would also account for:

• database row overhead;
• indexes;
• replication;
• metadata;
• backups;
• cache memory;
• operational headroom.

This is an upper planning envelope if all destinations are retained. Expiration can reduce live payload, but names are never reused: tombstones or equivalent reservations still grow with issuance. Alias length, ownership fields and idempotency records also consume space.

The 36.5 TB figure is URL payload only, in decimal units, using 365-day years. For illustration, 200 bytes per stored mapping including metadata and storage overhead would mean 73 TB for one copy, or 219 TB for three copies, before separate indexes, backups and headroom. These are planning assumptions, not measured row sizes.

Storage and throughput are separate constraints. Roughly 1.2K inserts per second does not by itself justify a complex distributed write architecture; retaining and operating hundreds of billions of records is a different challenge.

The read/write ratio also does not establish the cache hit rate. That depends on which links are requested and how often they are reused. If the measured hit rate were 95%, the average mapping-read load would be about 11,574 × 0.05 ≈ 579 reads/sec, excluding policy checks and consistency fallbacks. During a complete cache outage it could rise approximately twentyfold.

The important conclusion is that the mapping needs durable storage and an explicit model for cache misses, peak traffic and recovery.

3. API Design

The API exposes link creation, public resolution and administrative blocking. A request’s idempotency key belongs to the authenticated creator and identifies one creation operation.

Create a short URL

POST /api/v1/urls
Authorization: Bearer <access-token>
Idempotency-Key: launch-campaign-2026
Content-Type: application/json

{
  "longUrl": "https://example.com/articles/distributed-systems",
  "customAlias": "autumn-launch",
  "expiresAt": "2026-12-01T00:00:00Z"
}

Both customAlias and expiresAt are optional. Omitting the alias produces a generated code; omitting expiration creates a link without a scheduled end date. Validate the destination, alias grammar and future UTC timestamp before consuming allocator capacity.

HTTP/1.1 201 Created
Content-Type: application/json

{
  "shortUrl": "https://sho.rt/a/autumn-launch",
  "expiresAt": "2026-12-01T00:00:00Z"
}

Generated links use /{code}, for example /Ab91xK2. Custom aliases use /a/{alias}. Separating the routes prevents a customer-chosen name from reserving a future generated code. In this design aliases contain 3–64 lowercase ASCII letters, digits or hyphens. Reject noncanonical input instead of silently converting two submitted names into one.

Retrying the same idempotency key with the same payload returns the original result during the documented retention window. Reusing it with different input returns a conflict. The service responds only after the mapping and retry result are safely committed.

Resolve and block a link

Both public routes use the same resolver and policy checks:

GET /a/autumn-launch

HTTP/1.1 302 Found
Location: https://example.com/articles/distributed-systems
Cache-Control: no-store

An administrative POST /api/v1/links/block accepts the public path, requires a privileged identity and records an audit event. Its success acknowledges the durable status change; cache propagation remains subject to the stated takedown budget.

Error responses use Cache-Control: no-store in this design. Internal negative caching is handled separately. A database timeout is never evidence that a link does not exist. The distinction between unknown and permanently unavailable resources follows IETF RFC 9110.

4. High-Level Architecture

Creation establishes identity and durable state. Resolution retrieves that state and enforces the link’s lifecycle. Separate creator and resolver pools allow each workload to have its own concurrency, deployment and failure limits; they can still share a codebase.

flowchart LR
    C[Clients] --> G[Gateway and rate limits]
    G -->|Create| W[Creator pool]
    G -->|Resolve| R[Resolver pool]
    G -->|Authorized block| P[Policy handler]
    W -->|Reserve ID range| I[Durable allocator]
    W -->|Commit link and retry result| D[(Mapping store)]
    P -->|Update status| D
    P -.->|Invalidate| K[(Mapping cache)]
    R -->|Lookup| K
    R -->|Miss or refresh| D
    D -->|Record| R
    R -->|Populate| K
    R -->|Check status and expiry| R
    R -->|Redirect or policy response| C
    R -.->|Bounded event buffer| Q[Event stream]
    Q --> A[Analytics workers]

The mapping store includes authoritative writes and, where useful, read replicas. Its logical responsibilities stay the same as it is partitioned. The cache holds complete resolver records, including status and expiration. It does not independently fetch database rows: the resolver owns cache population.

Write path

The creator authenticates the caller, validates input and identifies the retry operation. It obtains an internal ID, chooses the generated or alias namespace, and commits the mapping with the idempotency result. A uniqueness constraint arbitrates alias races. Cache warming happens after commit and is best effort.

Read path

The resolver parses the route into a canonical lookup key, retrieves a fresh record and checks blocking and expiration. An eligible mapping produces a redirect; the other states produce explicit responses. Event capture uses a bounded local buffer and never waits for analytics processing.

This arrangement lets the service prioritize redirect availability without relaxing write correctness. During an allocator outage, existing links remain resolvable even when creation eventually exhausts its reserved ranges.

5. Short Code Generation Strategy

The central design question is:

How do we convert an identifier into a short string without collisions?

The short-code problem

Assume the short code can contain:

0-9
a-z
A-Z

That gives us:

10 + 26 + 26 = 62 characters

With an alphabet of 62 symbols, a code of length n provides:

62^n

possible values.

For example:

Our estimated lifetime capacity is approximately 365 billion URLs.

Six characters are therefore insufficient:

62^6 ≈ 56.8 billion

Seven characters provide substantial room:

62^7 ≈ 3.52 trillion

So seven characters provide enough address space if the allocated numeric IDs remain below 62^7. Dense allocation fits the projected count, but abandoned ranges, reserved values and IDs consumed by failed requests also use capacity. Monitor the allocation high-water mark.

Base62 has variable length unless padded. This design uses exactly seven characters, padding smaller encodings on the left with 0, and accepts only that canonical form. A full-width 64-bit identifier can require eleven Base62 characters; an arbitrary timestamp-based ID scheme does not automatically fit the seven-character budget. Never truncate an encoded unique ID to force it to fit.

Approach A: hash the URL

One possibility is to hash the original URL:

long URL
   ↓
hash function
   ↓
hash value
   ↓
short code

flowchart LR
    URL["Long URL"]
    HASH["Hash Function"]
    CODE["First N Characters"]
    DB[(Database)]

    URL --> HASH
    HASH --> CODE
    CODE --> DB

The problem is that a conventional cryptographic or checksum hash produces a much larger value than seven characters.

Truncating it introduces collisions.

For example:

URL A → abc1234
URL B → abc1234

The system must then detect the collision and generate another candidate.

A collision-resolution loop must use an atomic insert-if-absent operation:

for attempt in bounded_attempts:
    candidate = encode_to_code_space(hash(longUrl, attempt))
    if database.insert_if_absent(candidate, longUrl):
        return candidate
return allocation_error

A separate “check, then insert” is unsafe under concurrency. Each retry must change the candidate, and the database must arbitrate competing inserts. A Bloom filter can sometimes avoid lookups, but cannot replace that atomic uniqueness check.

Random candidates have the same collision-handling requirement. If all 365 billion issued links occupied generated codes, the seven-character space would be about 10.36% full, giving a uniformly drawn candidate a 10.36% collision probability at that point. Custom aliases occupy a separate namespace. This is different from asking whether any collision has ever occurred across the entire dataset.

Approach B: generate an ID and encode it

A cleaner approach is to assign every URL a globally unique numeric ID.

For example:

ID = 2,009,215,674,938

Then encode that number using Base62.

flowchart LR
    URL["Original URL"]
    ID["Unique Numeric ID"]
    B62["Base62 Encoder"]
    CODE["Short Code"]

    URL --> ID
    ID --> B62
    B62 --> CODE

This has an important property:

If the numeric ID is unique, the encoded short code is also unique.

There is no collision-resolution loop.

Base62 encoding

Base62 represents a number using 62 symbols.

One possible alphabet is:

0-9
a-z
A-Z

Conceptually, the encoding works like any other positional numeral system.

For example, suppose a numeric value is repeatedly divided by 62:

value / 62 → quotient + remainder

The remainder identifies one Base62 character.

Continue until the quotient reaches zero, then read the remainders in reverse order.

For a much smaller illustrative value:

11157

the resulting Base62 representation is:

2TX

The exact character sequence depends on the alphabet ordering chosen by the implementation. With the alphabet shown here, 11157 → 2TX is the unpadded representation; the seven-character public code would be 00002TX.

The key idea is not the specific representation; it is that Base62 gives us a compact textual representation of a much larger numeric identifier.

Hashing vs. ID-based encoding

For this design, ID generation + Base62 is the preferred approach.

The trade-off is that the ID generator becomes a critical infrastructure component.

Applied to the “Create URL” flow introduced in Section 4, the mechanism behind the “Generate short code” step looks like this:

ID:
2009215674938

Base62:
zn9edcu

Mapping:
zn9edcu
    ↓
https://example.com/articles/distributed-systems

The numeric ID is allocated first, encoded into zn9edcu using the Base62 scheme described above, and that code is what gets persisted and returned to the client.

6. Data Model, Write Path & Consistency

A record for each public link

The storage key represents the public address, including its namespace:

/Ab91xK2          → g:Ab91xK2
/a/autumn-launch → a:autumn-launch

This same key identifies the database record, cache entry and policy invalidation. An alias is another kind of public link, not a second address automatically created for every generated link. Both kinds receive an internal ID for analytics and administrative tracking.

A PostgreSQL representation is:

CREATE TABLE url_mapping (
    lookup_key TEXT COLLATE "C" PRIMARY KEY,
    id BIGINT NOT NULL UNIQUE CHECK (id >= 0),
    owner_id BIGINT NOT NULL,
    long_url TEXT,
    created_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
    expires_at TIMESTAMPTZ,
    status TEXT NOT NULL DEFAULT 'active'
        CHECK (status IN ('active', 'blocked', 'retired')),
    CHECK (status = 'retired' OR long_url IS NOT NULL),
    CHECK (expires_at IS NULL OR expires_at > created_at)
);

lookup_key is case-sensitive because generated Base62 codes distinguish uppercase and lowercase. The creator constructs the key from validated fields rather than accepting an arbitrary key from the client. Primary-key and unique constraints already create the corresponding PostgreSQL indexes. PostgreSQL: Constraints

The destination may be removed when an expired link is retired. Its row retains the public name, internal ID and expiration metadata as a tombstone. retired is a terminal state; an old bookmark must never begin resolving to a different destination. Compaction can reduce the retained record, but must preserve the namespace reservation.

An index on expiration can support cleanup batches. Its write and storage cost belongs in the capacity model. Cleanup reclaims payload; resolution enforces expiry independently. Database TTL is not an exact deadline mechanism: DynamoDB, for example, can remove expired items days later. AWS: Using time to live in DynamoDB

Creation, retries and competing aliases

sequenceDiagram
    participant C as Client
    participant W as Creator
    participant I as Local ID Pool
    participant D as Mapping Store

    C->>W: POST destination, optional alias and expiry, retry key
    W->>W: Authenticate and validate
    W->>D: Look up caller-scoped retry result
    alt Completed retry with matching payload
        D-->>W: Original result
    else New operation
        W->>I: Consume unique internal ID
        I-->>W: ID
        W->>W: Build g:code or a:alias key
        W->>D: Atomically insert mapping and retry result
        D-->>W: Committed result or conflict
    end
    W-->>C: Creation result or conflict response

The first retry lookup is an optimization. A transaction must still arbitrate concurrent requests for the same (owner_id, idempotency_key), compare their stored request payload and commit only one outcome. Losing transactions return the committed result for identical input; different payloads conflict. Requests for the same alias with different retry keys compete on lookup_key.

Keep the retry record and mapping in one transaction in the initial database deployment. Define a retention interval long enough for the supported retry behavior. After it expires, an old retry key cannot be assumed to identify the previous operation. AWS Builders’ Library: Making retries safe with idempotent APIs

Sharding requires an explicit continuation of this protocol. One approach is a durable request record, keyed by caller and retry key, that reserves an internal ID and target lookup key before attempting the mapping insert. Recovery resumes that same insert and completes the request record; it never generates a second identity. The target insertion is conditional, and an existing row is treated as a retry only when its operation identity matches. Until the protocol completes, a timeout represents an uncertain outcome, not permission to create another mapping.

Why destination deduplication is not the creation contract

Two campaigns may share a destination but have different aliases, owners and expiration times. They are distinct link resources. The normal creation path therefore does not search for an existing destination.

If storage analysis later justifies a separate destination dictionary, use a digest to locate candidates and compare the full URL before reusing stored payload. SHA-256 requires 32 binary bytes or 64 hexadecimal characters, and digest equality alone is not proof of URL equality. PostgreSQL: pgcrypto

Such payload deduplication must preserve each link’s identity and lifecycle. It also adds a lookup or cache dependency and reference-management work, so its value must be measured against the simpler self-contained record.

Why the ID generator matters

In a distributed system, multiple API servers may create URLs simultaneously:

API Server A → ?
API Server B → ?
API Server C → ?
API Server D → ?

They cannot safely use a local counter:

Server A: 1001
Server B: 1001   ← collision

We need IDs that are globally unique.

A production system could use a distributed ID-generation strategy based on:

• allocated ID ranges;
• database sequences;
• timestamp-based identifiers;
• dedicated ID-generation services.

One useful pattern is range allocation.

ID service
   │
   ├── Server A → IDs 1,000,000-1,999,999
   ├── Server B → IDs 2,000,000-2,999,999
   └── Server C → IDs 3,000,000-3,999,999

Each application server consumes IDs locally and replenishes its pool before exhaustion. The allocator must durably reserve disjoint ranges before handing them out. On restart, abandon uncertain unused IDs rather than replaying them; allocator failover and restored backups must never reissue a previously reserved range.

Range allocation makes application instances operationally replaceable, but the local pool is correctness-sensitive state. Overlapping process snapshots must not consume the same range. Existing pools allow creation during an allocator outage only until they are exhausted. The allocator’s high-water mark must include abandoned ranges as well as successful mappings.

An atomic counter is not a durable allocator

Redis INCRBY counter B returns the value after incrementing. If it returns H, the arithmetic interval is [H - B + 1, H]; it does not return the beginning of the range. Redis: INCRBY

Atomic execution prevents concurrent increments from overlapping on a running primary. Redis replication is asynchronous by default, and even WAIT does not establish strong consistency across failover. A rolled-back counter can therefore reissue previously handed-out values. Redis: Replication

Abandoned IDs create harmless gaps; reissued IDs threaten correctness. A uniqueness constraint can reject duplicates, but recovery still needs to restore a safe allocation frontier. Never overwrite an existing mapping on conflict. Keep range reservations in an authoritative store with the required failover guarantees; do not silently substitute an evictable cache counter. Test ambiguous reservation responses and allocator recovery explicitly.

Consistency considerations

The API must not return a new short URL before its mapping meets the chosen durability policy. Asynchronous replication can lose acknowledged writes during failover; synchronous replication trades additional coordination and latency for stronger durability. Durable replication and visibility to replica queries are separate guarantees. PostgreSQL: Log-Shipping Standby Servers

For immediate usability, a resolver that sees a replica miss should confirm absence against an authoritative, sufficiently consistent source before returning 404. Cache warming helps, but eviction or cache failure means it cannot establish this guarantee by itself. Return a temporary error if absence cannot be established during an outage.

This fallback needs protection against random-code scans. Validate syntax first, rate-limit abuse, and cache authoritative negative results only briefly. Never turn a replica miss or database timeout into a long-lived negative cache entry. A strongly consistent mapping store is an alternative if its cost and latency fit the workload.

7. Read Path & Caching

Resolution is a lookup followed by a policy decision. A cached destination is insufficient: the resolver needs status, expiration and a bound on how recently the mutable status was checked.

Cache-aside resolution

sequenceDiagram
    participant C as Client
    participant R as Resolver
    participant K as Mapping Cache
    participant D as Mapping Store

    C->>R: GET generated code or alias
    R->>R: Parse canonical namespaced key
    R->>K: Read record
    alt Fresh cache hit
        K-->>R: Record and policy deadline
    else Miss or policy freshness expired
        R->>D: Read record with required consistency
        D-->>R: Record or authoritative absence
        R->>K: Best-effort cache population
    end
    R->>R: Check status and expiration
    alt Active and unexpired
        R-->>C: 302 with Location and no-store
    else Blocked, expired or unknown
        R-->>C: 403, 410 or 404 with no-store
    end

Dependency failures take a separate path and return 503 if no sufficiently fresh record can be served. Simplified resolver logic is:

resolve(path):
    key = parse_canonical_route(path)
    record = cache.get_with_short_timeout(key)

    if record missing or now >= record.policy_fresh_until:
        record = read_with_required_consistency(key)
        if read unavailable:
            return 503
        if authoritative absence:
            return 404
        cache.best_effort_set(key, record)

    if record.status == blocked:
        return 403
    if record.status == retired:
        return 410
    if record.expires_at exists and now >= record.expires_at:
        return 410

    events.try_enqueue(record.id, request_metadata)
    return 302 with Location and Cache-Control: no-store

Blocked status takes precedence over expiration here; either outcome prevents a redirect. The actual handler applies the same HTTP no-store policy to errors and validates missing or malformed routes before accessing storage.

Expiration and policy freshness

Expiration is an immutable timestamp checked on every resolution, including cache hits. Keep clocks synchronized and define the accepted clock-error bound. Mutable status has a separate freshness deadline. A positive cache entry must not be served beyond:

usable_until = min(policy_fresh_until, expires_at or infinity)

An expired mapping can be fetched or cached as a terminal response, but it cannot produce a redirect. Refreshing from an arbitrarily lagging replica must not restart the policy deadline. Use an authoritative policy read or a known replication-lag bound, and preserve the original deadline when copying records between cache tiers.

The blocking handler updates authoritative status and publishes invalidations. TTL remains the fallback if delivery fails. Versioned invalidation, or a policy read bound to a request-start timestamp, is needed to prevent an in-flight stale read from repopulating the cache with a fresh lifetime after a block. Every layer, including an edge resolver, must honor the same maximum staleness budget.

Availability has a limit: when status freshness cannot be established within that budget, the service returns a temporary error instead of indefinitely serving a potentially blocked link.

301 vs. 302 redirects

301 Moved Permanently communicates a permanent redirect and permits heuristic caching. 302 Found communicates a temporary redirect, but it can still be cached when explicit freshness allows it. A 302 alone does not guarantee another request reaches the resolver. IETF RFC 9110: HTTP Semantics, Sections 15.4.2–15.4.3

The service sends Cache-Control: no-store so compliant HTTP caches do not retain the response. no-cache has a different meaning: storage is allowed, but reuse requires validation. These HTTP directives do not prohibit our application from caching mapping records internally. IETF RFC 9111: HTTP Caching, Section 5.2.2

If edge redirects are introduced, choose their response-cache policy explicitly and move policy checks and event capture to that layer as needed. Origin logs will not see requests served entirely at the edge. No choice of redirect status establishes exact human click counts: bots, previews and retries also generate requests.

Cache sizing and locality

A resolver cache entry contains the complete decision record:

Key: g:Ab91xK2 or a:autumn-launch
Value: id, long_url, status, expires_at, policy_fresh_until

Choose eviction based on measured reuse, such as an LRU or frequency-aware policy. Expiration controls eligibility; eviction controls memory consumption. A popular entry may stay resident while still requiring policy refreshes.

Size the cache from the working set, not from lifetime mappings. Measure entry overhead, hit rate, churn and the traffic that reaches storage during a rolling deployment or cold start. Include invalid-name scans in load tests: their access distribution is very different from repeated clicks on popular links.

Measure the complete lookup path

A memory-access timing is not a Redis-request timing. Network transit, connection handling, queueing, serialization and server execution all contribute to a distributed-cache lookup. A database can also serve frequently used pages from its own memory. Compare measured cache-hit and cache-miss latency distributions under representative concurrency, rather than treating raw RAM-versus-SSD timings as endpoint benchmarks.

For this service, track edge-to-response p99, cache hit rate, database fallback concurrency and hot-key skew together. Adding a cache is useful only if its latency and load benefits outweigh the additional dependency for the actual access pattern.

Handling hot URLs

Imagine a single link receives 500,000 requests per second. This is a stress scenario, not a measured capability of the proposed architecture. A cache hit prevents a database read, but a single cache key can itself become a bottleneck. Redis Cluster maps a key to one hash slot; adding shards does not distribute that key across all of them. Redis: Cluster specification

Possible responses include small per-process caches for popular links, deliberately replicated hot entries with suitable read routing, or edge resolution. Each copy must participate in the takedown policy. Benchmark the hot-key case across network, cache, API and analytics capacity.

Coalesce concurrent cache misses, stagger expirations, bound database concurrency, and shed excess load instead of letting a cold cache overwhelm storage. A cache outage can cause a service outage even though the durable mappings remain intact. AWS Builders’ Library: Caching challenges and strategies

8. Scaling, Reliability & Security

Database scaling

A single database may eventually become insufficient for storage capacity, write throughput, read throughput, or operational availability.

Replication. Read replicas can absorb database reads:

flowchart LR
    APP[Application Servers]

    PRIMARY[(Primary DB)]
    R1[(Read Replica 1)]
    R2[(Read Replica 2)]

    APP -->|Writes| PRIMARY
    PRIMARY -->|Replication| R1
    PRIMARY -->|Replication| R2

    APP -->|Reads| R1
    APP -->|Reads| R2

However, the cache should absorb most redirect traffic before it reaches the database.

Sharding. At larger scales, mappings can be distributed across multiple database partitions. To understand the partitioning trade-off, consider a numeric-ID hash:

hash(id) % N

or by ranges:

Shard 1 → IDs 0-99B
Shard 2 → IDs 100B-199B
Shard 3 → IDs 200B-299B
...

Range-based partitioning can be convenient for some operational workloads, while hash-based partitioning generally provides a more even distribution. The exact strategy depends on the database technology and operational requirements.

With roughly increasing allocation, static ID ranges concentrate new writes on the currently active range or ranges. Allocated blocks may be consumed out of order, but this does not guarantee balanced writes across all storage shards.

Switching to one active time bucket does not solve that throughput problem: all current writes still target the active bucket. Time buckets help lifecycle management; parallel write capacity requires multiple active partitions, for example a time bucket combined with a hash suffix. That pattern is illustrated in AWS: Using write sharding to distribute workloads evenly.

hash(id) % N is an introductory routing formula, not a complete resharding strategy. Changing N remaps many records. A production design can instead map IDs to a stable set of logical buckets and map those buckets to physical shards, with versioned routing and controlled data migration.

For this service, choose hash(lookup_key) as the input to logical-bucket routing. Both generated codes and aliases then identify their target shard without a directory lookup. Every request for a particular alias reaches the same authoritative partition, where uniqueness is enforced.

The internal ID still comes from the global allocator. A per-shard unique index on that ID is a local safeguard, not global allocation coordination. Retry records may route elsewhere, which is why the recoverable creation protocol in Section 6 matters. Hashing for shard placement does not conceal a sequential public code.

Stateless application servers

The API tier should not store session-specific state locally.

flowchart LR
    CLIENT[Clients]
    LB[Load Balancer]

    A1[API Server 1]
    A2[API Server 2]
    A3[API Server 3]

    CACHE[(Shared Cache)]
    DB[(Shared Database)]

    CLIENT --> LB

    LB --> A1
    LB --> A2
    LB --> A3

    A1 --> CACHE
    A2 --> CACHE
    A3 --> CACHE

    A1 --> DB
    A2 --> DB
    A3 --> DB

Any request can be routed to any healthy server. A server creating links needs its own safely allocated ID pool; redirect requests do not depend on that pool. If one server fails:

Server 1 ✕
Server 2 ✓
Server 3 ✓

the load balancer simply stops sending traffic to the failed instance. The API tier can then expand, provided shared dependencies have sufficient capacity:

more traffic
    ↓
more API instances

Scale creation and resolution independently when needed

The read and write paths can initially share a codebase and deployment. If creation bursts or background work compete with redirects, route POST /api/v1/urls to a creator pool and redirect requests to a resolver pool, with separate concurrency limits and scaling policies. This need not introduce separate repositories or a distributed workflow between the pools.

Creators require allocator access; resolvers require mapping and policy reads. Keeping those permissions and resource budgets separate limits the effects of a creation-side incident. Shared storage and cache dependencies still need capacity protection; splitting the API tier alone does not remove them.

Rate limiting and abuse prevention

A URL shortener is an attractive abuse target. Attackers can attempt to generate huge numbers of URLs, automate requests, create phishing links, distribute malware, exhaust identifier capacity, or overload the API.

A rate limiter should therefore sit in front of the creation endpoint.

flowchart LR
    CLIENT[Client]
    RL[Rate Limiter]
    LB[Load Balancer]
    API[API Servers]

    CLIENT --> RL
    RL -->|Allowed| LB
    RL -->|Rejected| BLOCK[429 Too Many Requests]
    LB --> API

Limits can be based on IP address, authenticated account, API key, geographic or network signals, reputation, or adaptive traffic thresholds.

The redirect endpoint may require different limits because legitimate links can suddenly receive enormous traffic.

Security considerations

A URL shortener also creates a security boundary between the short code and an arbitrary destination. The system should consider:

Malicious destinations. Validate destination suitability at creation and provide an abuse-reporting and takedown process. A link can become harmful after it is created. Arbitrary redirects can facilitate phishing; interstitial destination warnings may be useful for suspicious links. OWASP: Unvalidated Redirects and Forwards

SSRF. Returning a redirect does not itself fetch the destination from the server. Preview generation, crawlers and malware scanners do. Isolate those workers, restrict network egress, allow only intended schemes, and prevent access to internal, loopback, link-local and metadata addresses. Validate resolved IPv4/IPv6 addresses, address DNS rebinding, and either disable redirects or validate every hop. Syntax checks alone are insufficient. OWASP: Server-Side Request Forgery Prevention

Enumeration. Generated Base62 codes reveal an ordered identifier space. A reversible obfuscation can change their appearance without establishing confidentiality. The service therefore treats generated names and aliases as public, and enforces authentication separately wherever access to private resources is required.

There is a more basic limit here. From our own capacity assumptions:

365,000,000,000 / 62^7 ≈ 10.36% occupied

Even a perfect permutation of the namespace leaves that density unchanged. If all 365 billion issued links used generated codes, roughly one out of ten uniformly sampled codes would identify an issued name. Aliases reduce that generated-name occupancy; expired and blocked records further reduce the fraction that still redirects. Obfuscating sequence order cannot make this a private namespace.

Public links can use direct IDs with abuse monitoring and rate limits. If links must be confidential, use authorization or a separately designed high-entropy capability scheme, accepting longer codes. Identifier complexity is defense in depth, not a replacement for access checks. OWASP: Insecure Direct Object Reference Prevention

Input validation. Accept only absolute HTTP(S) URLs, impose a documented size limit, and reject malformed input and dangerous schemes such as javascript:. Use a URL parser and the HTTP framework’s header APIs. Do not place raw unvalidated input into response headers.

Revocation. Persist a blocked status, invalidate application and edge caches, and specify a maximum propagation delay. A failed invalidation must still be bounded by expiration or another policy check. Do not promise immediate revocation if cached status may remain active. The no-store response policy also avoids intentionally placing long-lived redirects in client caches.

Availability and failure modes

API server failure. Route around unhealthy instances and recover in-flight creations through their retry records. Abandon uncertain local ID ranges. The remaining instances need enough capacity to absorb the displaced traffic.

Cache failure. Mappings remain durable, but fallback traffic can overwhelm the database. Use bounded fallback concurrency and tested overload behavior; do not assume availability merely because the data still exists.

The cache should accelerate the system, not become the only copy of critical data.

Database replica failure. Traffic can be redirected to healthy replicas.

Primary database failure. A backup requires restoration before it can serve requests; it is not a live failover replica. Use failover with fencing of the old primary. Define recovery-time and recovery-point objectives, test backup restoration, and document whether acknowledged writes can be lost. Replica promotion alone does not establish zero data loss. Retain enough healthy replicas and capacity for the chosen single-node-failure target.

ID generator failure. This is more serious because new URLs cannot safely be created without unique IDs. One way to reduce the impact is to preallocate ID ranges:

API server
    ↓
local ID pool
    ↓
IDs available without network round trip

The application can continue creating URLs temporarily even if the central allocator is unavailable, provided its local pool has capacity.

9. Analytics, Evolution & Trade-offs

Analytics architecture

Analytics should not depend on synchronously updating an analytics database for every redirect. But asynchronous processing does not automatically guarantee delivery.

For approximate usage statistics, the resolver makes a bounded, non-blocking offer to an in-process event buffer. A background producer publishes events independently:

sequenceDiagram
    participant C as Client
    participant API as Redirect Service
    participant B as Bounded Local Buffer
    participant P as Background Producer
    participant Q as Event Queue
    participant A as Analytics Worker

    C->>API: GET generated code or alias
    API->>API: Resolve mapping and check status and expiry
    API->>B: Try enqueue without blocking
    B-->>API: Accepted or dropped if full
    API-->>C: 302 with Cache-Control no-store

    P->>B: Drain buffered events
    B-->>P: Event batch
    P->>Q: Publish batch with bounded retries
    Q-->>P: Acknowledge accepted batch
    Q->>A: Deliver events

The redirect does not wait for broker acknowledgement. Process crashes and full buffers can lose events. Put limits on queue memory, retry duration and background concurrency, and monitor drops so analytics failure cannot consume all redirect resources. The relative timing of background publication and the client receiving the response is not guaranteed by this diagram.

For stronger event durability, wait for durable acceptance before returning the redirect, or durably record the event for later publication. Either introduces storage or broker work into the acknowledgement path. Producer retries and consumer reprocessing may cause duplicates; assign event IDs and make aggregation idempotent where accurate counts matter. Kafka’s delivery guarantees depend on producer, broker and consumer behavior, and exactly-once effects in an external analytics store require cooperation from that store. Apache Kafka: Message Delivery Semantics

At the assumed 10:1 ratio, recording every eligible resolution would approach one billion events per day if nearly all requests redirect. With an illustrative 200-byte event, that is about 200 GB of raw events per day before replication and indexes. Use the internal link ID to aggregate generated links and aliases consistently; track blocked, expired and failed requests as separate operational metrics. Size retention and aggregation independently of the mapping store; avoid retaining detailed raw events indefinitely by default.

Report resolver requests separately from estimated human clicks. Define bot filtering, sampling and privacy controls, and minimize identifying telemetry. This article does not prescribe a jurisdiction-specific compliance policy.

What happens as the system grows?

The architecture can evolve incrementally. Early deployments preserve the link semantics but do not automatically satisfy the final availability and capacity targets:

• Stage 1, Small deployment: API → Database. Enough for a small product.
• Stage 2, Read-heavy workload: API → Cache → Database. Caching absorbs most redirect traffic.
• Stage 3, Multiple application instances: Load Balancer → API × N → Cache + Database. The web tier becomes horizontally scalable.
• Stage 4, Database replication: Primary → Read replicas. Database reads are distributed.
• Stage 5, Large-scale storage: API → Cache → Sharded database. Mappings are partitioned across multiple storage nodes.
• Stage 6, Global deployment: regional application clusters, geographically distributed caches, replicated databases, edge/CDN integration, and regional traffic routing.

For multiple regions, assign each logical bucket an authoritative write region or use a store that provides equivalent conflict arbitration. Two regions must not independently approve the same alias during a partition. ID ranges can be allocated to regions in advance, but that solves numeric allocation, not alias ownership or mapping replication.

A remote resolver also needs a path for newly created links and sufficiently fresh policy state. During a partition, creation may pause while eligible cached redirects continue within their freshness budget. Region failover requires fencing old writers and recovering outstanding request records before accepting conflicting writes. These constraints determine the topology more directly than the number of geographic replicas.

Key design trade-offs

There is no universally correct URL-shortener architecture. The important part is understanding why each component exists.

Hashing vs. generated IDs. Hashing can derive an identifier directly from the URL, but collisions must be handled. Generated IDs provide uniqueness naturally, but require distributed ID allocation. For a large-scale service, generated IDs plus Base62 provide a simpler creation path.

Redirect status and cache policy. Permanent redirects and caching can reduce resolver traffic. The service uses 302 plus no-store for request-level policy checks; edge caching requires a deliberate alternative policy and event-capture location.

Cache vs. database. The database remains authoritative. The cache exists to reduce latency and database load.

Identifier exposure and partitioning. Direct Base62 IDs are compact and predictable. Hash-based partitioning distributes storage load but does not hide public IDs. Confidentiality requires a separate access or capability design.

Event durability vs. redirect independence. Best-effort buffering protects redirect latency but can lose events. Durable event acceptance adds a dependency; asynchronous downstream processing alone does not remove it.

Operational validation

Validate the design through failures that cross component boundaries:

Monitor creator and resolver availability separately. Track p99 latency by cache outcome, alias conflicts, allocator reserve depth, replica lag, policy age and event loss. Those measurements expose whether the architecture meets its stated contract rather than merely containing the expected components.

Conclusion

A scalable URL shortener manages more than a dictionary of destinations. It gives every public name a stable identity, a lifecycle and a consistent resolution policy.

The design follows those responsibilities through the system: generated codes use durable ID allocation, aliases rely on atomic namespace reservation, retries preserve operation identity, and expiration is checked wherever a redirect can be issued. Caches accelerate retrieval while respecting policy freshness. Creator and resolver pools scale independently, and analytics has an explicit delivery contract.

The essential workload remains asymmetric:

Writes establish durable link identity; reads repeatedly retrieve that identity and enforce its current eligibility.

The resulting architecture can grow from an indexed database to partitioned storage and regional resolution without changing that contract. Its reliability comes from preserving the contract through concurrency, caching and failure, then testing those guarantees under the workload it is intended to serve.