Delivery and Retries

A worker can finish a report and still fail to acknowledge its message. Another worker can then receive the same work. Retry policy controls what the consumer requests after an attempt; it cannot make the handler’s effects and the broker’s state one atomic operation.

This page assumes the shared report job from Producing Jobs and a running consumer. A delivery is a message reserved for handling. Settlement is the consumer’s request to acknowledge, retry, reject, or terminally fail that delivery. Backend storage, retention, and native delivery limits still apply; compare Drivers.

Acceptance Boundaries

Successful dispatch confirms driver acceptance, not handler execution. A dispatch error after transmission starts can leave acceptance uncertain: the backend may have stored the message before its reply was lost. Repeating the facade call creates a new message ID and can enqueue another copy.

The consumer requests acknowledgement only after the handler returns nil. If an external effect succeeds before the handler returns an error, retry can repeat it. Even a successful handler can run again if acknowledgement fails or the reply is uncertain. Losing a lease can allow another worker to start while the first handler is still running.

There is no unconditional portable at-least-once delivery guarantee and no exactly-once handler guarantee. Memory loses pending work at stop or process exit. Broker retention, delivery limits, and retry-forwarding policy can remove work before completion. RabbitMQ’s retry confirmation covers its intermediate TTL queue, not later forwarding; its forwarding protections and expiry limit matter even when the work queue is durable.

Duplicate Effects

Use a stable business identity such as BuildReport.ReportID, not a transport ID, to make repeats harmless. Typed handlers do not receive the message ID, and the facade creates a fresh one on every dispatch. JetStream’s duplicate window or SQS FIFO deduplication does not deduplicate repeated facade payloads automatically.

For database-only work, enforce uniqueness and record completion in the same transaction as the effect. For a remote effect, use the provider’s idempotency key when available and reconcile uncertain outcomes before repeating the operation. A local flag written before the remote call can suppress work that never happened; a flag written afterward leaves a crash window for duplicate effects. Neither placement makes the remote call atomic.

Repeats and ordering are different concerns. Concurrent workers and retries can reorder effects even when each effect is idempotent. Enforce sequence or version checks in application state if order matters. A durable outbox protects dispatch intent across database commits but still needs duplicate-safe consumption.

Handler Outcomes

Return the result that matches what the application knows, not what you hope the broker will do. The consumer converts it into a settlement request:

For an upstream report service that returns a rate-limit delay, your handler can override backoff. Here upstream.Build and rateLimited belong to your integration; the latter recognizes an error with a usable response.RetryAfter duration:

response, err := upstream.Build(ctx, payload.ReportID)
if rateLimited(err) {
	return queue.RetryAfter(err, response.RetryAfter)
}

Handle the remaining result normally. The explicit delay does not bypass the attempt limit. Negative delays become zero, selecting driver timing; positive delays are lower-bound requests that a driver can round up or reject, not silently shorten. Backend availability and retention still determine whether delivery occurs.

If loading a report shows that its source record no longer exists, retrying cannot repair that absence. With errors imported and your application’s reports.ErrNotFound sentinel, classify that branch:

if errors.Is(err, reports.ErrNotFound) {
	return queue.Permanent(err)
}

Permanent preserves the cause and matches queue.ErrPermanent under errors.Is. Check the operation’s error before wrapping it: Permanent(nil) is still a non-nil terminal outcome, not success. RetryAfter and Reject also accept nil causes while retaining their explicit outcome.

Rejection is deliberately backend-specific. Prefer Permanent for irreparable application failures unless you intentionally configured native rejection/redrive:

These are successful settlement paths, not unconditional guarantees after an RPC error. The facade exposes no portable dead-letter inspection or replay API.

Retry Policy

Set attempts and backoff on a fresh job declaration, using queue from arkoren.dev/core/service/queue and time. This alternative report definition allows one initial run and four retries:

var BuildReportJob = queue.NewJob[BuildReport]("build-report", func(job *queue.JobBuilder) {
	job.OnQueue("reports")
	job.Retry(5)
	job.Backoff(2*time.Second, 2, 10*time.Second)
})

The four retries request delays of 2, 4, 8, and 10 seconds. Retry(n) counts the first attempt. Zero, the default, imposes no facade cap; native caps and retention can still end delivery. All shipped drivers report attempt counts. A custom driver reporting zero cannot support facade attempt-limit enforcement.

Backoff takes an initial delay, multiplier, and maximum. A multiplier below one or NaN becomes one; a zero maximum removes the cap. Negative durations are invalid. Without backoff, the driver’s default timing applies: memory can retry immediately without a native attempt cap, so always choose a finite policy for work that may keep failing.

For a custom curve, use an alternative construction with BackoffFunc:

var BuildReportJob = queue.NewJob[BuildReport]("build-report", func(job *queue.JobBuilder) {
	job.OnQueue("reports")
	job.Retry(4)
	job.BackoffFunc(func(attempt int) time.Duration {
		return time.Duration(attempt-1) * 5 * time.Second
	})
})

The callback receives the attempt about to run, at least two, so these retries request 5, 10, then 15 seconds. A negative result becomes zero; a nil callback restores driver timing. It replaces any earlier Backoff, and vice versa. The callback’s captured state remains shared and must support concurrent use.

Check driver bounds before choosing delays. RabbitMQ buckets delays and has TTL/forwarding limits; NATS uses delayed negative acknowledgement; SQS rounds up to seconds and allows at most 12 hours subject to remaining visibility; Cloudflare rounds up to seconds and allows at most 24 hours. Unsupported-delay settlement can fail even though the application correctly requested a retry.

Execution Deadlines

Set job.Timeout(duration) in the definition callback to bound one attempt’s context. Zero means no deadline. Execution is synchronous: passing the deadline does not force a return. If the handler returns an error after the deadline, normal outcome and attempt policy apply. If it returns nil, the consumer still acknowledges success. Panics are recovered as ordinary errors with stack text and follow retry/cap policy; avoid secrets in panic values and logs.

Execution timeout is not the broker’s reservation deadline. NATS renews every half AckWait; SQS renews every half visibility duration, with a 12-hour limit from the receive request. Failed renewal can permit concurrent redelivery. RabbitMQ uses unacknowledged deliveries rather than a renewable lease; connection loss can release unsettled work. Cloudflare has no renewal: budget for batch waiting, execution, and settlement within visibility, and make execution honor its shorter timeout. Increasing batch size does not increase concurrency within one consume call.

Shutdown Settlement

When the consumer’s context ends, an active handler receives cancellation. If it returns an error matching that context error, such as ctx.Err() or a %w wrapper, the consumer requests retry with zero delay, bypassing the immediate attempt-cap and backoff checks. A different error follows ordinary policy. The reported delivery count is not discounted: a later ordinary failure can therefore exhaust the job’s cap. Job timeout alone does not select this shutdown exception.

After decoding and execution finish, settlement gets a fresh 30-second context detached from consumer cancellation, retaining context values. Unknown names receive that same fresh budget without execution. Acknowledgement for completed work can therefore finish after shutdown begins, but the context cannot keep a closed connection alive or make a non-cooperative SDK interruptible.

Cancel and join consumers before stopping drivers or closing borrowed clients. Allow shutdown time for both handler completion and settlement. If the application budget expires first, driver cleanup can interrupt settlement. In memory, canceling a consumer preserves its unadmitted backlog; stopping the driver instead discards pending work and prevents a final retry from being enqueued.

An unsuccessful settlement is returned to the driver, not converted into successful retry or failure. RabbitMQ ends consumption and closes its channel to release unsettled jobs. NATS, SQS, and Cloudflare log the error and can keep consuming while unsettled work becomes eligible for redelivery. A settlement error does not prove that the backend ignored the request.

Poison Messages

A poison message cannot complete with the current handler or envelope. A payload that fails typed JSON decoding requests terminal failure immediately; another attempt cannot repair those bytes. Drivers also handle malformed native envelopes before the typed handler is called, using their own failure path. Do not expect facade events for a message the facade never receives.

An unknown job name may instead mean an older worker received newly deployed work. The consumer requests a 30-second retry until the reported delivery count reaches ten, then requests terminal failure with queue.ErrUnknownName. For a longer deployment window, construct a consumer with a different cap:

consumer := queue.NewConsumer(driver, func(worker *queue.ConsumerBuilder) {
	worker.UnknownNameAttempts(20)
})

Zero restores ten; a negative value removes the facade cap, not broker limits. The cap relies on reported attempts. Native delivery-limit exhaustion, such as JetStream MaxDeliver, does not itself invoke the facade’s terminal failure path or create its failure record. Deploy compatible workers first and investigate unknown-name failures instead of using unlimited retries as deployment policy.

Failure Records

Configure archives on Drivers before relying on terminal jobs being retained. RabbitMQ, NATS, and SQS can transfer a failure envelope to a configured destination before acknowledging/deleting the source. Publication failure leaves the source unsettled. Transfer and source acknowledgement are not atomic: a lost reply or crash can duplicate work or failure records. Archive access and retention must protect job data, raw malformed bodies, and unsanitized error text.

With an event bus configured, queue.JobFailed is emitted only after the driver accepts terminal settlement. It does not prove archive durability, undo remote effects, or indicate a native dead-letter destination. Rejection emits its own event rather than JobFailed; failed settlement emits no accepted-outcome event. Monitoring shows event wiring and how to distinguish handler telemetry from settlement evidence.

See the queue API.