50,000 Scheduled Tasks/sec
Sustained peak request volume during high-traffic events.
InfrastructureStaff Level
Distributed Job Scheduler & Orchestrator (Temporal/Quartz)
Execute millions of recurring cron and delayed background jobs reliably with worker heartbeats and distributed locking.
Target Scale
Engineering Scale & Performance SLAs
Target production parameters expected in a senior or staff interview round.
10,000 Multi-Tenant Enterprise Accounts
Daily active users generating read and write operations.
100 GB/day job execution logs
Projected data ingestion and replication storage capacity.
Job execution delay < 100ms
Strict end-to-end percentile latency SLA constraint.
Stage 01
Functional & Non-Functional Requirements
Establish clear problem boundaries before proposing architectural components.
Core System Capabilities
- Schedule one-time delayed jobs and recurring cron jobs across distributed workers.
- Support parent-child job workflows with conditional branches and retries with backoff.
- Worker heartbeat monitoring, dynamic task re-assignment, and cancellation.
- Searchable audit logs and live dashboard of job states and execution durations.
Reliability & Latency SLAs
- At-least-once execution guarantee (no job dropped due to worker crash).
- High timing accuracy: execute within 100ms of target execution epoch.
- Horizontal scalability to handle millions of concurrent pending delay timers.
Stage 02
Capacity Estimation Math
Step-by-step arithmetic conversions for QPS, storage, and bandwidth.
| Dimension | Calculation Formula | Estimated Result |
|---|---|---|
| Peak Job Dispatch Throughput | 100M jobs/day / 86,400s * 3x top-of-the-hour spike factor | 3,472 jobs dispatched/second |
| Delayed Job Timer Queue Sizing | 10 Million future-scheduled jobs waiting in storage | 10,000,000 active scheduled timer keys |
| Worker Heartbeat Telemetry Ingress | 5,000 workers sending health pings every 5 seconds | 1,000 heartbeat evaluations/sec |
Stage 03
Multi-Tier Architecture & Component Topology
How requests navigate ingress gateways, application logic, caching, and persistence.
Job API Gateway · Cron Expression Parser · Idempotency Manager
Validate job payload contracts, compute initial trigger timestamps, and persist to task database.
Hierarchical Timing Wheel · Redis Sorted Sets (ZSET by Epoch) · Timer Scanner Fleet
Maintain sorted delay timers, scan for jobs where `trigger_epoch <= now()`, and push ready jobs to dispatch queues.
Kafka / RabbitMQ Dispatch Topic · Worker Consumer Pool · Distributed Lock (Redlock)
Deliver tasks to assigned worker instances, prevent duplicate concurrent runs, and track lease expirations.
Heartbeat Monitor · Lease Renewal Daemon · Failure Recovery Worker
Reclaim jobs from dead workers whose heartbeats have expired and re-queue for execution.
Stage 04
Database Schemas & Partitioning Strategy
Entity models, indexing, and primary key partitioning.
PK: job_id
- job_id (UUID PK)
- tenant_id (UUID)
- job_type (VARCHAR(64))
- cron_expression (VARCHAR(64) NULL)
- next_run_at (TIMESTAMP)
- status (VARCHAR(32))
- retry_count (INT)
Composite index on (status, next_run_at) for efficient timer poller range queries.
PK: execution_id
- execution_id (UUID PK)
- job_id (UUID)
- worker_id (VARCHAR(64))
- started_at (TIMESTAMP)
- finished_at (TIMESTAMP NULL)
- exit_code (INT)
- logs_s3_url (VARCHAR(512))
Index on (job_id, started_at) for execution history and debugging.
PK: worker_id
- worker_id (VARCHAR(64) PK)
- queue_name (VARCHAR(64))
- last_heartbeat_at (TIMESTAMP)
- is_healthy (BOOLEAN)
Index on last_heartbeat_at for reaper daemon dead-worker detection.
Stage 05
Critical Architectural Trade-Offs
How to defend engineering compromises when challenged by interviewers.
Timer Implementation: Database Polling vs In-Memory Hierarchical Timing Wheel
Option A: Polling relational database `WHERE next_run_at <= NOW()` every second
Option B: Hierarchical Timing Wheel (Kafka/Netty style) backed by Redis ZSET
Rationale: Database polling causes massive table locking and index thrashing at scale. Hierarchical timing wheels provide O(1) insert and O(1) expire operations in memory.
Execution Semantics: Exactly-Once vs At-Least-Once with Idempotency
Option A: Strict Exactly-Once Execution
Option B: At-Least-Once Execution with Application Idempotency Keys
Rationale: True distributed exactly-once execution across network partitions is impossible due to the Two Generals problem. At-least-once with client idempotency prevents duplicate side-effects reliably.
Technical FAQ
Frequently Asked Questions: Distributed Job Scheduler & Orchestrator (Temporal/Quartz)
Key interview questions and conceptual defenses.
How do you prevent two worker nodes from executing the same job at the same time?
Workers must acquire a distributed lock with a fencing token (or an atomic conditional update `status = RUNNING WHERE status = PENDING`) before starting execution.
What happens if a worker server crashes mid-job without reporting an error?
The worker's heartbeat lease expires. The reaper daemon detects the missing ping, marks the execution as FAILED, and re-submits the job to the queue.
Simulate this architecture
Download ClawPad