Architecture at 500K Writes/Min: Dissecting X’s Write Path, Stateless Timers, and Dual-Adjacency Graphs

Designing a system to handle 500,000 posts per minute alongside tens of millions of active feed queries per second forces you to throw traditional architectural patterns out the window. At this tier of throughput, ACID guarantees break down, database secondary indices become lethal bottlenecks, and background cron tasks for expiring state collapse under operating overhead.

To sustain this volume, systems like X (formerly Twitter) separate synchronous write confirmation from asynchronous fan-out processing, rely on masterless LSM-tree storage engines, and use query-optimized data models tailored strictly to access patterns.

Here is the exact blueprint for handling high-volume ingestion, nested threading without secondary indices, stateless poll expirations, and dual-adjacency social graph storage.

1. System Topology & Dual-Path Ingestion

At high scale, a monolithic write path is an anti-pattern. Systems cannot afford to block client write requests while updating search indices, pushing notifications, and computing analytics.

X splits ingestion into two discrete paths immediately after edge authentication: a Synchronous Write Path optimized for sub-15ms client confirmations and an Asynchronous Event Stream for downstream fan-out.

Architectural Breakdown

Ingestion & ID Generation: Incoming requests hit an Envoy Gateway for TLS termination, authentication, and rate-limiting. Envoy routes requests over gRPC to the Post Service, which assigns a globally unique, k-sortable Snowflake ID to the payload.
Synchronous Persistence: The metadata payload (text, user ID, media references) writes synchronously to a masterless wide-column store (Cassandra/Manhattan) with a strict SLA of <15ms.
Async Event Streaming: Concurrently, an event is emitted to Apache Kafka (post-events topic, partitioned by post_id) within <2ms. Downstream consumers—including real-time search indexing, feed fan-out, safety processing, and analytics ingestion—consume off this bus without degrading the write path.
Blob Storage Isolation: Posts containing rich media do not send binary data through the transactional database. Large media assets are uploaded directly to Object Storage (S3 / GCP / X Blobstore) via pre-signed URLs, served through Cloudflare CDN, and referenced in the post payload purely by URI.

2. DB Storage Mechanics: Why Cassandra?

Choosing relational databases with single-leader replication models (e.g., standard PostgreSQL) for high-write feeds introduces severe master I/O bottlenecks.

X relies on wide-column stores like Cassandra or Manhattan for primary metadata persistence due to three core architectural decisions:

LSM-Tree Storage Engine: Writes are appended sequentially to an in-memory MemTable and flushed to immutable SSTables on disk. Append-only I/O turns random disk writes into sequential disk I/O, yielding write performance comparable to writing straight to RAM.
Masterless Multi-Node Writes: Any node in the cluster can accept a write request for any partition key. There is no single master node acting as a write bottleneck.
Tunable Consistency over ACID: Feeds do not require strict linearizability. Achieving Eventual Consistency (e.g., LOCAL_QUORUM reads/writes) ensures that a 100ms propagation delay to a peer in a different region does not block the client thread.

Stroring Comments on a Post: Text, Images, Videos

There is no need to create separate schema for comments, replies and quotes . Comments, replies, and quote posts are treated internally as standardized posts only containing embedded lineage metadata:

{
  "post_id": "9900112233",
  "author_id": "usr_bob",
  "text": "Great system design breakdown!",
  "reply_metadata": {
    "parent_post_id": "181920394820",
    "root_post_id": "181920394820"
  },
  "created_at": 1722506400
}

Loading All Comments for a Post or Individual Comment

Comments can be nested as well in nature, one comment on a post can also have other comments. To query specific comment that has multiple sub comments, indexing is not a right way for Cassandra. Querying comments using a secondary index (e.g., SELECT * FROM posts WHERE parent_post_id = X) forces a distributed database to perform a scatter-gather query across every node in the cluster. At scale, this causes severe tail-latency amplification and node timeouts.

The Pattern: Partition-Key Thread Routing

Comments use explicit schema partitioning. Replies are written to tables partitioned directly by root_post_id. Fetching an entire comment thread under a viral post requires hitting one single partition, eliminating cluster-wide scatter-gather operations entirely.

3. Expiring Polls After 24 Hours and Refreshing Live Results to Users

X provides options to host a poll that gather user votes for 24 hours expire after 24 hours. After 24 hours options to vote gets disabled and final results are shown. So how does it expire poll after 24 hours? Any background task or cron job? When hosting feature sets like 24-hour polls or live vote tallies, classic polling and scheduled jobs are not the right choice. Consider so many people hosting polls and so many background threads running to check timer of each and every poll, it’s not scalable at this huge user base. X uses stateless time validation.

Stateless Time Validation vs. Background Timers

Running background cron jobs or individual timers to "close" millions of active polls after 24 hours is horribly inefficient.

Instead, X uses Stateless Time Validation:

Every poll payload contains a created_at Unix timestamp.
The expiration logic is computed dynamically at the edge during reads or write-attempts:
IsExpired = CurrentTime - Timestamp > 86400
If valid, the submission proceeds; if invalid, the client UI disables the submit action directly. Zero background compute required.

Vote Processing: Write-Behind Cache & WebSocket Debouncing

As soon as user clicks on votes on X, it doesn’t write to main DB immediately. Directly incrementing atomic counters in Cassandra for viral polls creates disk row contention and compaction storms. The platform delegates volatile counts to an in-memory cache like - Redis and use write-behind cache technique paired with a debounced WebSocket distribution layer.

Architectural Decisions:

In-Memory Atomic Increments: Votes hit Redis nodes via non-blocking atomic commands (HINCRBY). Redis handles hundreds of thousands of increments per second per node.
Write-Behind Cache Persistence: Asynchronous background workers pull aggregated counter totals from Redis every few seconds and execute batched mutations to Cassandra for permanent state storage.

Updating Live Poll Results on Live Screen of User

Consider a viral poll post from celebrity that has reached to millions of people, every second 5K votes are getting clicked. 10 thousand people have this poll on their screen live. How does it update this live votes and also refresh to the screen of the users having poll opened?

Using Web Sockets: As soon as user open X, it opens a websocket connection for two way communication. That web socket connection helps to receive all the events and process the updated data. So as soon as the user goes to a poll, it sends a subscribe event to the already opened web socket and starts getting updated data. Now whenever user scroll past the poll then it sends unsubscribe event.
Any kind of such notifications like - typing notification in DM or new n notifications handled via same web socket connection
Now millions of users who have X poll active in their feed need to see the updated results. For viral posts, millions of users can’t keep making requests to servers to get the latest results. So to achieve that, X publishes "VoteUpdated" event to Pub/Sub kafka events. Clients subscribed to events of particular poll results via web socket can get the latest updates
For a viral poll, emitting a socket event for every single vote on a post with 5,000 votes/sec will instantly drop client connections and saturate network interfaces. So X uses aggregator workers that batch votes over a 500ms sliding window, emitting at most 1 aggregated snapshot frame per second per client stream. This drops server-side WebSocket overhead by over 99%.

4. Handling User Profile and Follower / Following Query Asymmetry

User profile is handled via separate micro service that uses graph DB internally to store the followers and following lists. Social relationships represent directed graph edges. The primary architectural challenge is that Followers and Following queries exhibit opposite access patterns with drastically different scale factors.

Forward Query: "Who is User X following?" (Executed during feed generation; bounded size, typically 10^1 - 10^3 edges).
Inverted Query: "Who follows User X?" (Executed during post fan-out; unbounded size, up to 10^7 - 10^8 edges for high-profile accounts).

Querying a single normalized table (e.g., SELECT followee_id FROM follows WHERE follower_id = ?) creates severe disk I/O bottlenecks because one direction will always require scanning non-indexed keys across distributed nodes.

Dual-Adjacency List Strategy

To resolve this asymmetry, the system maintains two separate, query-optimized adjacency tables (backed by graph-optimized key-value stores like Redis or custom engines like FlockDB):

1. Forward Graph (Query: Who am I following?)
   Key: user_id:123:following -> Set [456, 789, 1011]

2. Inverted Graph (Query: Who are my followers?)
   Key: user_id:123:followers -> Set [202, 303, 404, ...]

graph LR
    subgraph Mutate Action: User 123 Follows User 456
        Direction[Client API] -->|Dual Write Engine| ForwardGraph
        Direction -->|Dual Write Engine| InvertedGraph

        ForwardGraph["Forward Graph Table
(following_by_user)
Key: user_123"] -->|Appends| SetA["Set: {456}"]
        InvertedGraph["Inverted Graph Table
(followers_by_user)
Key: user_456"] -->|Appends| SetB["Set: {123}"]
    End

High-Efficiency Filtering: Mutes & Blocks

Unlike follow relationships, Mute and Block operations are low-volume, sparse directed edges.

Muted Users: Stored as simple sets (SET mute:user_123 -> {user_456}).
Blocked Users: Stored as bi-directional rules enforced at read time.

Because these sets are small ($<1,000$ entries for $99.9%$ of users), they are serialized into Bloom Filters or lightweight Redis Sets directly loaded into the active user session cache, enabling $O(1)$ in-memory filtering during feed compilation.

Architectural Trade-off Matrix

Architecture Pattern	Primary Advantage	Operational Cost / Trade-off	Bottleneck at Scale	Recommended Data Store
LSM-Tree Metadata Store	Sub-15ms write latencies; massive horizontal scalability.	Lacks ACID transactions; eventual consistency requires handling out-of-order reads.	High read amplification if compaction fails to keep pace.	Cassandra / Manhattan
Write-Behind Poll Caching	Protects persistent storage from atomic counter hot-spotting.	Risk of volatile metric loss (e.g., Redis node failover before DB flush).	In-memory cache memory saturation during global events.	Redis In-Memory Cluster
Debounced Sockets (500ms)	Cuts network throughput and frame processing overhead by $>99%$.	Intentionally introduces up to 500ms UI latency for live metrics updates.	WebSocket connection memory allocation (C10M connection problem).	Netty / Custom Erlang WS Tier
Dual-Adjacency Graph	$O(1)$ lookup performance for both Forward and Inverted graph queries.	Doubles write-path work; potential split-brain divergence between tables.	Distributed lock contention during dual-writes if not managed asynchronously.	FlockDB / Redis Sets

Production Failures

1. The Secondary Index Trap in Distributed NoSQL

X is not only a high write throughput system but also requires massive read scale
Anti-Pattern: Adding a secondary index on a wide-column store to query posts by tag or parent comment ID or generating user feed
Production Failure: As partitions multiply across hundreds of nodes, a single query using a secondary index queries every single storage node. Tail latency ($p99.9$) spikes from 10ms to over 5,000ms, triggering node starvation cascades.
Fix: Enforce single-partition queries via explicit schema design. Denormalize data and write duplicate records to partition keys mapped specifically to the read target. Also separate out read and write DB to handles both the scales separately

2. WebSocket Connection Storms on Viral Fan-out

Anti-Pattern: Subscribing every client connected to a active viral poll to an unthrottled real-time messaging pipeline.
Production Failure: A breaking global post causes 2,000,000 clients to open WebSockets simultaneously. Broadcasting every increment pushes billions of messages/sec, burning through server socket descriptors and causing memory exhaustion crashes across the WebSocket layer.
Fix: Apply server-side dynamic rate debouncing. Enforce aggressive maximum client push frames (e.g., maximum 1 Hz update rate per client thread) regardless of underlying event ingest volume.

3. Dual-Write Graph Divergence

Whenever user A follows user B, following list of user A needs to be updated as well as followers list of user B also needs to be updated.
Anti-Pattern: Synchronously updating the following table and followers table within application code without an isolation tier.
Production Failure: If the network connection drops halfway through executing dual writes, the system enters a permanently inconsistent state (User A shows they follow User B, but User B's follower count does not reflect User A).
Fix: Use an event-driven transactional outbox pattern. Write the graph intent to a single persistent log, and allow a dedicated CDC (Change Data Capture) worker pipeline to mutate the dual-adjacency tables asynchronously with retry guarantees.

5. Creating user feed and showing viral posts to millions

Going to explain in next article -

How does X ensure that what posts should be visible to your feed?
How does it handle generic posts of someone having 10 followers vs someone having a million followers
How does it show a viral post to so many new people
Even if I am not following someone, still his/her posts are shown to my feed. How?

So stay tuned.

Cheers,

Tech Builder

X Decoded: The Engineering Architecture Handling 500K Tweets/Min