Fencepost: Engineering a Distributed Job Scheduler Around Invariants, Failure, and Evidence
Distributed scheduling is often introduced as a timing problem.
Store a CRON expression, identify due jobs, and dispatch them to workers.
That description is accurate in the same way that describing a database as a place that stores rows is accurate. It captures the visible function while missing the engineering problem.
A production distributed scheduler is a coordination system. It must determine who has authority, how that authority expires, how stale owners are rejected, how work survives partial failure, and which guarantees can be defended when the scheduler interacts with systems outside its own transactional boundary.
I built Fencepost, a cloud-agnostic distributed job scheduler implemented with Java 21, Spring Boot, PostgreSQL and RabbitMQ, to explore those concerns directly.
The implementation currently includes:
Twelve Maven modules
Approximately 12,800 lines of source and test code
194 passing unit and integration tests
Database-backed leader election
Partition-based scheduler ownership
Fencing tokens
Transactional outbox dispatch
Worker execution leases
Durable retries
Dead-letter handling
Reconciliation
Tenant isolation
Audit logging
Local multi-node execution through Docker Compose
Repository:
The important part of the project is not the number of modules or tests. Those are supporting facts.
The important part is the set of architectural claims the system makes, the invariants that support those claims, the trade-offs deliberately accepted, and the evidence still required before the platform can be considered operationally proven.
The engineering problem is authority under partial failure
A distributed scheduler must remain correct when different participants have inconsistent views of the system.
A scheduler node may believe it owns a partition after its lease has expired.
A worker may believe it still owns an execution after another worker has taken over.
The database may contain a committed execution while the message broker contains no corresponding message.
The broker may deliver a message more than once.
An external service may commit a business side effect while the scheduler records no completion.
These are not unusual corner cases. They are the expected consequences of partial failure.
The architecture must therefore answer four fundamental questions:
Who has authority now?
How does that authority expire?
How are previous owners prevented from acting?
How does the system recover when a transition is only partially completed?
Fencepost addresses those questions using leases, fencing tokens, conditional state transitions, durable execution records, a transactional outbox and reconciliation.
1. Begin with invariants, not components
A common architecture process starts with components:
API
Scheduler
Database
Queue
Worker
That is useful, but insufficient.
Components describe structure. Invariants describe what must remain true when the structure is under stress.
For Fencepost, the most important invariants are:
One logical execution per scheduled occurrence
For a recurring job, the logical execution identity is derived from:
job ID + scheduled fire timeEven if multiple scheduler nodes observe the same due job during a rebalance or lease transition, the database must accept only one logical execution.
This is enforced through a durable uniqueness constraint, not an in-memory check.
Only the current ownership epoch may perform protected writes
Every lease acquisition increments a monotonically increasing fencing token.
A node holding token 41 cannot perform protected writes after another node acquires token 42.
Terminal execution states cannot move backwards
An execution that is completed, permanently failed, cancelled or dead-lettered cannot return to a running state through an ordinary state transition.
Dispatch intent is committed atomically with execution creation
The execution record and the outbox event representing the intent to dispatch must be stored in the same database transaction.
Retry eligibility is durable
The decision to retry, the next retry time and the attempt count must survive process restarts and broker outages.
Heartbeats do not establish authority
A heartbeat represents recently observed liveness. It does not prove that a node owns a partition or execution.
These invariants are more important than any specific framework or cloud service.
They define what the implementation must preserve if RabbitMQ is replaced by SQS, Docker Compose is replaced by ECS, or PostgreSQL is moved from a local container to a managed service.
2. State the execution guarantee honestly
Fencepost provides:
At-least-once execution with idempotency support.
It does not claim exactly-once execution.
That decision is not a limitation of RabbitMQ or PostgreSQL. It follows from the end-to-end system boundary.
Consider the following sequence:
1. A worker sends a payment request.
2. The payment service commits the payment.
3. The worker crashes.
4. Fencepost never records completion.
5. The execution lease expires.
6. Another worker retries the same execution.The scheduler cannot know whether the external side effect succeeded.
The scheduler database and an arbitrary external system do not participate in a single atomic transaction. Therefore, the scheduler cannot guarantee exactly-once business effects.
Fencepost instead creates a stable idempotency key.
For scheduled executions:
job ID + scheduled fire timeFor manual triggers:
job ID + caller-provided idempotency keyThe handler must propagate that key to the downstream system or maintain its own durable deduplication record.
This results in a more defensible contract:
Messages may be delivered more than once.
Dispatch events may be published more than once.
An execution may be retried after worker failure.
Scheduler state remains consistent.
Business-side duplication is prevented through handler-level idempotency.
Principal-level architecture requires more than selecting a delivery guarantee. It requires making the ownership of that guarantee explicit.
The scheduler owns durable execution state and delivery retry.
The handler owns idempotency of external effects.
The downstream service owns enforcement of its idempotency contract where supported.
3. Separate the authority domains
Fencepost separates three different ownership problems.
Cluster leadership
One scheduler node holds a cluster leadership lease.
The leader performs cluster-wide responsibilities such as:
Reconciliation
Node-health evaluation
Coordination
Maintenance activities
Leadership represents control-plane authority.
Scheduler partition ownership
The job space is divided into 128 logical partitions.
Multiple scheduler nodes own different partitions concurrently.
Scheduler A: partitions 0–42
Scheduler B: partitions 43–85
Scheduler C: partitions 86–127Partition ownership represents scheduling data-plane authority.
Execution ownership
A worker must acquire a separate lease before executing a particular job occurrence.
Execution ownership represents worker-plane authority.
These three questions must not be conflated:
Who may coordinate the cluster?
Who may create executions for this job partition?
Who may run and complete this execution?A single leader could process all schedules, but that would centralise throughput and increase failover impact.
A broker consumer could be treated as the execution owner, but that would confuse message delivery with durable authority.
A heartbeat could be treated as proof of ownership, but that would allow stale nodes to act after a lease transition.
Separating these domains makes the architecture easier to reason about and allows each one to scale and fail independently.
4. Heartbeat, lease and fence are different concepts
Fencepost uses the following model:
Heartbeat = observed livenessLease = temporary ownershipFencing token = authority epoch
This distinction is central to the correctness model.
Heartbeat
A heartbeat answers:
Did the process report recently?
What version is running?
Is the process active, draining or offline?
What capacity is it currently reporting?
A missed heartbeat is evidence of possible failure.
It is not sufficient evidence that ownership can safely be transferred.
Lease
A lease provides time-bounded ownership.
Partition: 27
Owner: scheduler-node-2
Lease expiry: 14:05:15 UTCWhen the lease expires, another node may attempt acquisition.
Fencing token
A lease expiry does not physically stop the previous process.
A JVM can pause because of:
Garbage collection
CPU starvation
Host suspension
Container-runtime interruption
Network isolation
Database connectivity loss
Suppose scheduler A acquires partition 27 with fencing token 41.
It then pauses longer than its lease.
Scheduler B acquires the same partition with fencing token 42.
When scheduler A resumes, it may still believe it owns the partition. Its writes must therefore include token 41 and be rejected.
The fencing token identifies the ownership generation.
Token 41: previous authority epoch
Token 42: current authority epochThis is stronger than an in-memory ownership flag and different from ordinary optimistic locking.
5. Optimistic locking protects data; fencing protects authority
Optimistic locking usually protects a record from concurrent modification.
UPDATE job_execution
SET status = 'COMPLETED',
version = version + 1
WHERE execution_id = :executionId
AND version = :expectedVersion;This prevents two transactions from silently overwriting the same version.
It does not, by itself, prove that the caller is still authorised to perform the update.
A stale worker may resume, reload the latest record version and attempt to complete work it no longer owns.
Fencepost therefore validates both record state and ownership epoch.
UPDATE job_execution
SET status = 'COMPLETED'
WHERE execution_id = :executionId
AND status = 'RUNNING'
AND worker_id = :workerId
AND fencing_token = :fencingToken;The distinction is:
Version:
Has the record changed?
Fencing token:
Does this actor still have authority?Both may be required.
This is representative of a broader distributed-systems principle: data concurrency and authority concurrency are related but not equivalent.
6. Partitioning distributes scheduling without centralising throughput
Fencepost maps jobs to fixed logical partitions:
partition = hash(job ID) mod 128Each active scheduler node attempts to own an approximately equal number of partitions.
The partition-management cycle performs three operations:
Renew existing partition leases.
Voluntarily release partitions above the node’s current target.
Acquire available or expired partitions.
This creates a decentralised and convergent rebalancing model.
There is no separate coordinator maintaining a full partition-assignment plan. Nodes gradually move the cluster towards balance.
Why fixed partitions were selected
Small and understandable coordination surface
Straightforward local testing
Predictable ownership records
No dependency on an external coordination platform
Sufficient parallelism for the current expected cluster size
Accepted limitations
Equal partition counts do not imply equal workloads.
Hot partitions may create uneven scheduler utilisation.
Changing the partition count requires migration.
Rebalancing is convergent rather than instantaneous.
Fixed partitions eventually limit horizontal scale.
Revisit signals
The design should be reconsidered when:
Partition load becomes materially skewed.
A single partition limits scheduling throughput.
The number of scheduler nodes approaches the effective partition count.
Online repartitioning becomes a business requirement.
Rebalance time fails the recovery objective.
Possible future approaches include:
Virtual partitions
Weighted partition ownership
Consistent hashing
Workload-aware allocation
Partition splitting
The decision is not that 128 is universally correct.
The decision is that fixed partitions are the simplest design that meets current requirements, and the system has explicit signals for when that assumption no longer holds.
7. The transactional outbox resolves the database-to-broker gap
The boundary between PostgreSQL and RabbitMQ creates a dual-write problem.
A naïve scheduling transaction might:
1. Insert the execution.
2. Publish to RabbitMQ.
3. Advance the job’s next fire time.This creates ambiguous failure windows.
Database succeeds; publication fails
The execution exists, but no worker receives it.
Publication succeeds; database commit fails
The worker may receive an execution that the scheduler does not consider committed.
Publication succeeds; acknowledgement is lost
The publisher retries and produces a duplicate.
Fencepost uses a transactional outbox.
Within one PostgreSQL transaction, the scheduler:
BEGIN TRANSACTION
INSERT job_execution
INSERT outbox_event
UPDATE job_definition.next_fire_time
COMMITThe execution and the intent to publish are committed atomically.
A separate publisher reads pending outbox events and publishes them to RabbitMQ.
PostgreSQL
|
| committed execution + outbox event
v
Outbox publisher
|
v
RabbitMQ
|
v
WorkerIf RabbitMQ is unavailable, the outbox retains the event.
If the publisher crashes after sending but before updating the outbox record, the message may be published again.
That is acceptable because the execution model already assumes duplicate delivery.
The outbox guarantee is not exactly-once publication.
It is:
A committed execution will not be permanently lost because the application failed between its database transaction and broker publication.
8. Message receipt is not execution ownership
RabbitMQ delivery is a transport event.
It is not proof that the receiving worker is the only process permitted to execute the job.
A message may be redelivered because:
The consumer disconnected.
An acknowledgement was lost.
The message was negatively acknowledged.
The broker recovered.
The publisher emitted a duplicate.
Fencepost workers therefore acquire a database-backed execution lease before invoking a handler.
Receive message
-> load execution
-> validate eligibility
-> acquire execution lease
-> transition to RUNNING
-> execute handler
-> persist outcome
-> release lease
-> acknowledge messageThe execution lease contains:
Execution ID
Worker identity
Attempt number
Expiry time
Fencing token
If worker A pauses and loses the lease, worker B can acquire a newer fencing token.
Worker A cannot later overwrite execution state using its stale token.
This protects scheduler state from stale completion.
It does not prove that worker A produced no external side effect before losing ownership. That remains an end-to-end idempotency concern.
9. Retries are part of the scheduler’s domain
Retries are sometimes implemented entirely through message-broker features.
That is convenient, but it weakens the platform’s ability to explain and govern execution behaviour.
A scheduler should be able to answer:
Why was this execution retried?
When will it run again?
How many attempts have occurred?
Which errors are retryable?
What is the maximum retry age?
Why was the execution dead-lettered?
Can an operator replay it safely?
Fencepost stores retry state in PostgreSQL.
The retry calculation uses exponential backoff:
delay = base delay × 2^(attempt - 1)with jitter to prevent synchronised retry storms.
Failures are classified into:
Retryable
Non-retryable
Retry exhausted
Exhausted and non-retryable executions are moved to a dead-letter table with failure context.
The broker transports execution messages.
The scheduler database determines whether an execution is eligible to run.
This keeps retry semantics stable across messaging implementations.
RabbitMQ, SQS and Kafka differ significantly in redelivery and visibility behaviour. Fencepost does not require those transports to expose identical semantics because the authoritative retry decision remains in the scheduler.
10. Reconciliation is a primary runtime capability
Distributed systems continuously produce incomplete transitions.
Examples include:
An execution remains RUNNING after its worker disappears.
An outbox event remains pending after the publisher crashes.
A scheduler node remains registered after its process stops.
An execution lease expires without a terminal state.
A retry time passes while dispatch is unavailable.
Fencepost includes a leader-only reconciliation loop that checks for:
Expired execution leases
Stuck outbox events
Long-running executions without valid ownership
Stale scheduler nodes
A recovery decision is based on durable evidence.
Execution state: RUNNING
Execution lease: expired
Worker heartbeat: staleThe execution may be transitioned to failure and evaluated for retry.
Reconciliation must be idempotent.
Running the same sweep more than once must not create:
Duplicate retries
Duplicate dead-letter records
Invalid terminal-state transitions
Overwritten successful completions
Leader-only reconciliation reduces concurrent recovery attempts, but it is not the correctness mechanism.
The recovery operations themselves must remain safe under repetition.
This is an important platform-design principle:
Reconciliation should not compensate for an unreliable core model. It should repair the incomplete transitions that remain unavoidable even in a correct model.
11. Graceful shutdown is an optimisation, not a guarantee
During a controlled shutdown, Fencepost:
Stops accepting new work
Marks the node as shutting down
Allows in-flight executions to drain
Releases partition leases
Releases execution leases
Marks the node offline
This improves rolling deployments and reduces recovery delay.
However, the architecture cannot depend on graceful shutdown.
Processes may be terminated without warning. Hosts may disappear. Network access may be unavailable during shutdown.
Correctness must continue to depend on:
Lease expiry
Fencing
Durable state
Reconciliation
Explicit lease release improves recovery time. It does not establish the underlying safety guarantee.
This distinction is important in reviewing operational mechanisms.
A mechanism that improves the common path should not be mistaken for the mechanism that preserves correctness in the worst path.
12. Cloud agnosticism is enforced through architecture
Fencepost runs locally using:
Java 21
Spring Boot
PostgreSQL 16
RabbitMQ 3
Docker Compose
AWS is a deployment target, not part of the correctness model.
The architecture follows ports and adapters:
scheduler-domain
^
scheduler-application
^
infrastructure adapters
|- PostgreSQL
|- RabbitMQ
|- REST API
|- future SQS
|- future S3The core modules do not depend on:
AWS SDKs
Spring persistence annotations
RabbitMQ client types
Container-orchestration APIs
Cloud metadata services
Architecture tests scan the modules for prohibited imports.
This makes the boundary executable.
Cloud agnosticism does not mean pretending all providers behave the same.
RabbitMQ and SQS differ in:
Acknowledgement semantics
Visibility timeouts
Redelivery behaviour
Ordering guarantees
Dead-letter handling
Operational scaling
The application remains portable because scheduler correctness does not depend on any one broker providing:
Exactly-once delivery
Global ordering
Durable retry scheduling
Exclusive consumer ownership
Adapters translate provider behaviour into a stable application contract.
13. Platform architecture includes tenancy, governance and operations
A scheduler that supports multiple teams or business domains becomes a platform, not merely an application.
That changes the engineering concerns.
The platform must define:
Tenant boundaries
Resource ownership
Handler registration
Scheduling quotas
Retry limits
Payload restrictions
Operational responsibilities
Audit requirements
Retention policies
Upgrade compatibility
Fencepost currently includes tenant-scoped queries and API-key-based authentication.
Tenant identity flows through:
API key
-> security principal
-> tenant context
-> application service
-> tenant-scoped repositoryThis design was chosen for portability and testability.
The accepted risk is that every query must correctly include the tenant predicate.
For higher-assurance environments, the architecture may require defence in depth:
PostgreSQL row-level security
Separate schemas
Separate databases for sensitive tenants
Per-tenant encryption keys
Cross-tenant security tests
Tenant-aware rate and concurrency limits
The broader lesson is that platform isolation cannot be added only at the API gateway.
It must be reflected in every durable access path.
14. Audit availability and audit durability are separate decisions
Fencepost records audit events for mutating operations.
The current audit service is non-blocking: an audit write failure does not fail the primary business operation.
That decision favours scheduler availability.
It may not be appropriate in every environment.
For an internal platform, best-effort audit may be sufficient.
For a regulated system, the requirement may be:
No state mutation may be committed without a corresponding durable audit event.
That would require audit data to be:
Written in the same transaction
Added through a transactional outbox
Delivered to an immutable store
Monitored for lag
Protected by retention and access controls
Principal and Distinguished Engineers must clarify which property matters more in a given context:
Business-operation availability
Audit completeness
Regulatory evidence
Recovery simplicity
The correct architecture depends on the consequence of losing either the operation or its audit record.
15. Mechanism completeness is not operational evidence
Fencepost implements many mechanisms expected in a production scheduler:
Durable schedules
Leader election
Partition ownership
Fencing
Transactional outbox
Worker leases
Durable retries
Dead-letter handling
Reconciliation
Graceful shutdown
Tenant isolation
Auditing
Retention
That does not yet prove the system meets its operational targets.
The current non-functional requirements include targets such as:
Objective | Target |
Fire time to dispatch lag | Under 5 seconds at p99 |
Leader failover | Under 30 seconds |
Partition rebalancing | Under 60 seconds |
Graceful shutdown drain | Under 30 seconds |
These are targets, not published benchmark results.
The project has not yet completed:
Load testing against the NFRs
Long-running soak testing
Full telemetry instrumentation
Structured logging
Published chaos-test measurements
AWS deployment
Multi-Availability-Zone validation
A production-readiness assessment should separate at least four levels.
Level 1: Mechanism implemented
The code contains the expected pattern.
Level 2: Correctness tested
Automated tests validate important success and failure cases.
Level 3: Operational objective measured
The mechanism satisfies defined latency, throughput and recovery targets under representative conditions.
Level 4: Operational capability institutionalised
Dashboards, alerts, runbooks, ownership, capacity planning and incident procedures are established.
Many systems are labelled production ready at Level 1 or Level 2.
A Principal or Distinguished Engineer should make the evidence level explicit.
16. The next engineering phase is evidence production
The most valuable next work for Fencepost is not adding more patterns.
It is proving the existing architecture.
Required metrics
The platform should measure:
Scheduled fire time to execution creation
Execution creation to outbox publication
Outbox publication to worker receipt
Worker receipt to lease acquisition
Leader-election duration
Partition-rebalance duration
Execution-lease expiry and takeover
Oldest pending outbox event
Retry volume
Dead-letter volume
Conditional-update conflicts
Database lock wait
Per-partition execution lag
These measurements should decompose total scheduling delay rather than expose only one end-to-end number.
For example:
Total dispatch lag
= scheduler scan delay
+ database transaction time
+ outbox publication delay
+ broker delivery delayWithout decomposition, an SLO violation identifies a symptom but not the failing subsystem.
Required load scenarios
The platform should be tested with:
Large numbers of active jobs
Sustained due-job rates
Burst workloads
High concurrent execution
Slow handlers
Retry storms
Hot partitions
Broker outages
Database connection exhaustion
Reconciliation backlog
Simultaneous node restarts
Required soak tests
A 24–72-hour test should monitor:
Memory growth
Thread leakage
Database connection leakage
Lease churn
Leadership churn
Partition imbalance
Outbox growth
Retry accumulation
Execution lag
Retention effectiveness
Required recovery tests
Evidence should be published for:
Leader process termination
Long JVM pause
Scheduler network isolation
Worker termination
RabbitMQ outage
PostgreSQL restart
Rolling deployment
Full cluster restart
Backup restoration
Future Availability Zone failure
This work converts architectural intent into operational confidence.
17. Architecture decisions require revisit conditions
An architecture decision is incomplete if it records only the selected option.
A stronger decision statement is:
We selected X because of these constraints, accepted these risks, and will revisit the decision when this evidence changes.
Fencepost’s key decisions can be expressed in that form.
PostgreSQL as the coordination store
Why selected
Already required for durable state
Supports conditional atomic updates
Easy to run locally
Avoids another distributed dependency
Keeps the architecture understandable
Risks accepted
Coordination traffic consumes database capacity
Database failure affects several control-plane functions
High lease churn may create contention
Failover performance depends on database recovery
Revisit when
Coordination traffic affects execution latency
Lock wait becomes material
Failover objectives cannot be met
Cluster size exceeds practical PostgreSQL coordination capacity
Fixed 128-partition model
Why selected
Predictable
Easy to reason about
Simple ownership schema
Sufficient for the expected initial node count
Risks accepted
Hot partitions
Uneven workload
Difficult partition-count migration
Eventual scale limit
Revisit when
Partition load becomes significantly skewed
One partition limits throughput
Scheduler-node count approaches partition parallelism
Online partition scaling becomes necessary
Leader-only reconciliation
Why selected
Avoids concurrent full-cluster sweeps
Simplifies recovery reasoning
Reuses existing leadership
Risks accepted
Reconciliation pauses during failover
The leader performs additional workload
Slow sweeps can delay recovery
Revisit when
Sweep duration approaches the reconciliation interval
Leader capacity becomes constrained
Recovery objectives require parallel reconciliation
RabbitMQ as the initial broker
Why selected
Easy local operation
Explicit acknowledgement model
Good developer visibility
Suitable for Docker Compose
No cloud account required
Risks accepted
Additional operational component
Different semantics from future SQS deployment
Broker scaling and high availability must be managed
Revisit when
AWS deployment becomes the dominant runtime
Operational ownership of RabbitMQ becomes costly
Queue scale or durability requirements justify a managed alternative
Application-layer tenant isolation
Why selected
Portable
Easy to test
Straightforward repository design
No database-specific security dependency
Risks accepted
Every query must include tenant scope
A missed predicate can become a security defect
Database administrators retain broad access
Revisit when
Regulatory requirements demand stronger isolation
High-risk tenants require dedicated boundaries
Repository complexity makes manual enforcement unreliable
This decision model demonstrates architecture as a continuously managed system of assumptions rather than a static diagram.
18. Principal and Distinguished Engineer responsibilities extend beyond the codebase
At senior engineering levels, the question is not only whether the scheduler works.
The questions include:
Who should use the platform?
Which workloads should not use it?
What is the onboarding model?
What guarantees are part of the platform contract?
How are incompatible handlers prevented?
How is capacity allocated between tenants?
Who owns failed executions?
Which teams respond to dependency failures?
How are schema and message formats evolved?
What is the migration path from existing schedulers?
How is operational cost attributed?
When should a team use a workflow engine instead?
A scheduler platform needs explicit boundaries.
Fencepost is suitable for independent, time-triggered units of work that can tolerate at-least-once delivery and implement idempotency.
It should not automatically evolve into:
A general workflow engine
A distributed transaction manager
A human-approval platform
A real-time stream processor
A substitute for event-driven domain logic
Those systems have different state, dependency and compensation models.
A platform becomes difficult to operate when every adjacent requirement is added to the same control plane.
Distinguished-level architecture often involves saying not only what the platform can do, but what it must deliberately refuse to become.
19. Evolution must preserve compatibility
A scheduler stores future intent.
That creates a longer compatibility horizon than many stateless APIs.
A job created today may execute months later. A queued message may survive deployment. A retry may occur after a handler version changes.
The platform therefore needs compatibility rules for:
Job-definition schema
Execution-message schema
Handler versions
Payload schema
Database migrations
Rolling deployments
Retry behaviour
Dead-letter replay
Fencepost uses schema-versioned execution messages.
The broader evolution model should include:
Backward-readable message formats
Explicit handler versioning
Expand-and-contract database migrations
Compatibility tests across adjacent releases
Replay tests for old dead-letter messages
Deployment policies for mixed application versions
The key principle is:
The scheduler must not assume that every durable record and queued message was produced by the currently running version.
This becomes increasingly important as the scheduler is adopted by multiple teams.
20. AI accelerates implementation but not accountability
AI can generate much of the visible implementation:
Controllers
Database entities
Migration scripts
RabbitMQ configuration
Retry utilities
Docker Compose
Test scaffolding
Metrics instrumentation
It can also recommend patterns such as leases, fencing and outboxes.
The differentiating engineering work is deciding:
Which failure modes matter for the business
Which guarantees can be defended
Where ownership boundaries belong
What must be fenced
Which retries are safe
Where idempotency responsibility sits
What evidence is required
When a decision must be revisited
Which complexity should not be introduced
AI can propose alternatives and accelerate implementation.
It cannot assume accountability for the consequences of the architecture.
When code generation becomes cheaper, the value of engineering judgment increases.
The scarce capability becomes the ability to connect:
business consequence
-> system invariant
-> architectural mechanism
-> operational signal
-> recovery procedureThat is the level at which Principal and Distinguished Engineers are expected to operate.
21. The deeper lesson
Distributed scheduling is not primarily about CRON.
It is about authority, durability, recovery and evidence.
A dependable scheduler must answer:
Which node has authority now?
How does another node prove that authority has expired?
How are stale owners rejected?
How does committed work survive broker failure?
How are duplicate deliveries made safe?
How are incomplete transitions discovered?
Which system is authoritative for retry?
Where does idempotency responsibility sit?
Which guarantees are real?
What evidence supports those guarantees?
What signals would invalidate the current architecture?
Fencepost addresses these questions through:
Database-backed leases
Monotonically increasing fencing tokens
Separate leadership, partition and execution ownership
Conditional state transitions
Transactional outbox dispatch
Worker execution leases
Stable idempotency keys
Durable retry schedules
Dead-letter records
Leader-driven reconciliation
Enforced architectural boundaries
It runs locally through Docker Compose and keeps cloud dependencies outside the core modules. AWS is a future deployment target rather than part of the correctness model.
The implementation is substantial, but the next stage is more important: producing evidence that the system meets its stated objectives under sustained load and realistic failure.
That is the distinction between building a distributed system and engineering a dependable platform.
The implementation creates the mechanisms.
The invariants explain why they should work.
The tests validate important scenarios.
The telemetry, benchmarks, failure exercises and operating model determine whether the platform can be trusted.
Comments