When two workers reach for one job.

Weave is still in development. This note describes the push-worker design, not a claim of production scale.

The race

A worker claims a notification and sends it to a provider. Before it records success, the request times out. Another run can eventually pick up that notification. Did the provider deliver the first attempt? The database cannot know just from a lost response.

I separate two questions: who may update the job, and whether a user has already received the message. Solving the first does not automatically solve the second.

A lease is temporary ownership

1. Claim. A batch is associated with a lease token and an expiry.
2. Renew. While processing, keep ownership alive within the run’s time budget.
3. Check. Skip work if its lease is no longer ours.
4. Finish conditionally. Update the row only if the current lease token still matches.

The conditional finish is the important boundary. A slow worker cannot overwrite the result of the worker that took over. A final update without that ownership condition would reintroduce the race.

Test the overlap, not just each worker

The useful test runs two workers on a fake clock. Advance time beyond a lease boundary, let another run take ownership, then allow the first run to finish. Its final write must leave the second worker’s state untouched.

I also classify 429 and server errors as temporary and stop starting new rows when the execution budget is spent. These choices make retries and interrupted runs explicit parts of the lifecycle.

The cost of this design

Lease renewal adds database work. Short leases make recovery faster but increase the chance of overlapping runs; long leases delay recovery after a crash. The right duration depends on measured provider latency and runtime limits, so I would tune it with telemetry rather than treat a constant as a guarantee.

This is the standard I want in my systems: the failure path should have an explanation as clear as the successful one.