Imagine telling a personal assistant: "Remind my coworker to inspect the production database at 9:00 AM three weeks from today."

If your assistant writes that note on a random piece of paper and loses it under a coffee mug, your team misses a critical update. Now, multiply that scenario by millions of users scheduling messages across thousands of companies, all expecting delivery down to the exact second.

How do you build a system that guarantees a message scheduled three months ago pops up on a user's screen at the right moment—even if half your server infrastructure crashes the night before?

Let's break down the system architecture behind Slack's scheduled messaging pipeline using simple, real-world concepts.

Vitess is a database management system built on top of MySQL. It acts like an intelligent receptionist standing in front of dozens of standard MySQL databases. Instead of your app needing to know which database holds which company's data, Vitess automatically routes your query to the right place behind the scenes.

Slack handles up to 10,000 scheduled messages every single second. A single, traditional database server would instantly choke under this volume of concurrent write operations. Vitess breaks up the massive load by sharding (splitting up) the data across multiple database instances based on workspace_id. To the rest of the application, it still looks and feels like one single database.

Running sharded databases makes complex cross-workspace queries very difficult. You can't easily join data across two different companies because their records live on physically separate servers.

Instead of saving human-readable date strings like "August 16th, 2026 at 09:00 AM EDT", the database stores schedule execution targets as Epoch Timestamps - a single, plain number representing the exact number of seconds elapsed since January 1, 1970 (e.g., 1770000000).

Computers struggle with timezones, daylight saving changes, and localized date strings. Comparing two text strings or converting timezones on the fly burns unnecessary CPU power. Comparing two numbers (1770000000 <= 1770000005) takes fractions of a nanosecond, making time comparisons lightning-fast.

Epoch timestamps are completely unreadable to human engineers during debugging. You always need helper tools to translate raw timestamps back into readable dates when inspecting logs.

A background service that runs once every hour to look ahead at the main database, locate all messages scheduled to go out within the next 24 hours, and load (hydrate) them into distributed cache.

If a user schedules a message 3 months in advance, keeping that payload idling in cache for 90 days wastes money and storage. Keeping it safely stored on disk in the main database costs almost nothing. We only move data into cache right before it is needed.

If this look-ahead worker crashes or fails silently, upcoming scheduled messages won't get moved into memory on time, causing delivery delays unless a backup mechanism catches them.

All messages need to be scheduled in next 24 hours need to be kept in cache. But there can be 1000 messages scheduled per second so how to organise those in cache for easy access? A Sorted Set (ZSET) is a unique data structure inside Redis where every item is assigned a numerical score. Redis automatically keeps all items strictly sorted by this score at all times.

In Slack's system:

Instead of scanning millions of records to find what needs to be sent right now, workers can ask Redis: "Give me all message IDs where the score is less than or equal to the current timestamp." Because the list is pre-sorted, finding the right messages takes almost zero effort.

RAM is expensive and volatile. If the Redis server loses power and lacks proper backup snapshots, anything stored exclusively in memory vanishes.

Instead of traditional cron jobs that spin up once a minute, these are persistent worker programs running constantly inside RAM. Every 250 milliseconds, these workers execute a short, embedded database script called an Atomic Lua Script inside Redis.

When you have multiple worker servers pulling jobs off the same queue simultaneously, two workers might grab the same message at the exact same millisecond and send it twice. An atomic Lua script guarantees that fetching the message ID and instantly deleting it from the queue happens as one un-interruptible action. No two workers can ever claim the same job.

If a Lua script contains a bug or takes too long to run, it completely freezes the Redis instance because Redis runs scripts on a single thread.

A WebSocket connection is a persistent two-way highway held open between your app and Slack's servers, allowing messages to pop up instantly without refreshing. Client-Side Idempotency means the Slack app on your phone or computer checks every incoming message against its own local database (using SQLite) to ensure it hasn't already processed that specific ID.

Network connections in the real world drop constantly. If a server sends a message to your phone, but your phone's Wi-Fi cuts out before sending an acknowledgment back, the server might try sending it again. Checking IDs locally on the receiving device ensures you never see duplicate messages on your screen.

The client device must maintain its own local tracking storage, which increases app memory and battery usage slightly.

Assume you have millions of messages in cache and suddenly system crashed. System took 5 mins to recover and once boot up it brought all scheduled messages for next 24 hours in cache. But what about messages that were scheduled during the crash window? Or what if something is missed to be sent from cache? A background service scans the primary database once every 60 seconds looking for orphaned messages - scheduled items whose target delivery time has passed, but are still marked as SCHEDULED instead of SENT.

If a Redis server crashes, a network cable gets unplugged, or a worker node dies mid-task, messages can get dropped. The sweeper acts as a fallback system to ensure zero message loss. It scans a safety window from 24 hours ago up to NOW() - 1 minute.

The 1-minute buffer is intentional: it prevents the sweeper from competing with active real-time workers processing live messages right now.

Querying large database tables for un-sent messages can slow down your database if you don't build specialized database indexes covering both status and scheduled_time.

This diagram illustrates how a scheduled message request is accepted, saved to the database, and selectively moved into high-speed memory if it is due soon.

This diagram shows how workers continuously extract tasks from fast memory and deliver them safely to the client app without duplicate deliveries.

Cheers,

Tech Builder