Recurring Durable Executions

Motivation

Many workflows need to run repeatedly — nightly reports, daily invoice generation, cache warmers, heartbeat checks, ETL pipelines. A naive approach uses a cron job that calls a script, but this breaks down when:

The script crashes midway through a multi-step process (invoice sent, but DB not updated)
You need visibility into past runs (did the report actually complete?)
Steps within the workflow need retries with checkpointing
Overlapping runs could cause duplicate side effects (double-charging customers, sending duplicate emails)

Recurring Durable Executions combine cron-like scheduling with Zart’s durable execution engine. Each tick starts a new DurableExecution that survives crashes, checkpoints each step, and respects overlap policies to prevent concurrent run chaos.

What Are Recurring Durable Executions?

A recurring durable execution combines the scheduling power of a cron or fixed-delay ticker with the full reliability of Zart’s durable execution engine. Each tick starts a new DurableExecution handler — with step checkpointing, retries, admin visibility, and result persistence.

When to use recurring durable vs. plain recurring `ScheduledTask`

Concern	Plain `ScheduledTask`	Recurring Durable Execution
Step-level fault tolerance	No	Yes — each step is checkpointed
Automatic retries	Manual	Yes — via `RetryConfig`
Admin / UI visibility	Limited	Full execution record
Parallel step support	No	Yes
Appropriate for	Lightweight fire-and-forget	Multi-step workflows that must not be lost

Use a plain recurring ScheduledTask (see Recurring Scheduler Tasks) for lightweight jobs where a single function call is sufficient. Use recurring durable executions when you need steps, retries, and observability.

Quick Start

use zart::prelude::*;
use zart::OverlapPolicy;
use zart_scheduler::Recurrence;
use serde_json::json;

// 1. Implement your handler as usual
struct NightlyReport;

#[async_trait::async_trait]
impl DurableExecution for NightlyReport {
    type Data = serde_json::Value;
    type Output = serde_json::Value;

    async fn run(&self, data: Self::Data) -> Result<Self::Output, TaskError> {
        zart::require(FetchInventoryStep).await?;
        zart::require(WriteReportStep).await?;
        Ok(json!({ "status": "ok" }))
    }
}

// 2. Register and schedule in one call
let worker = WorkerBuilder::from_backend(&pg)
    .register_durable_task("nightly-report", NightlyReport)
    .register_recurring_durable::<NightlyReport>(
        "nightly-report",                          // task_id (unique per worker)
        "report-{occurrence}",                     // execution ID template
        // Fixed delay for local dev / testing:
        Recurrence::FixedDelay { duration_ms: 5_000 },
        // Cron for production:
        // Recurrence::Cron { expression: "0 2 * * *".into(), timezone: "UTC".into() },
        OverlapPolicy::SkipIfRunning,              // overlap policy (see below)
        json!({ "warehouse": "EU-1" }),            // initial payload
    )
    .build();

Overlap Policies

When a new tick fires while a previous execution is still running, the OverlapPolicy controls what happens.

Policy	Running execution	New occurrence	Canonical use case
`SkipIfRunning`	Continues to run	Silently skipped	Batch reports, ETL pipelines
`CancelAndRestart`	Cancelled immediately	Fresh execution starts	Config refresh, cache warming
`AlwaysStart`	Continues to run	Also started	Independent audit windows, per-slot ingestion

`SkipIfRunning` (recommended default)

The new tick is a no-op if any execution derived from the same template is currently in a Scheduled or Running state. Choose this when running the current instance to completion is more valuable than starting a fresh one.

`CancelAndRestart`

The running execution is cancelled via DurableScheduler::cancel and a new one is started with the next occurrence ID. Choose this when you always want the latest state and partial work from the old run is undesirable.

`AlwaysStart`

Every tick unconditionally creates a new execution, even if one is already running. Choose this when occurrences are fully independent (e.g. time-windowed audit logs that must all complete regardless of overlap).

Execution ID Templating

The id_template parameter supports one substitution token:

Token	Replaced with
`{occurrence}`	The 0-based occurrence counter, incremented on every successful dispatch

Examples:

"report-{occurrence}" → "report-0", "report-1", "report-2", …
"audit-{occurrence}-eu" → "audit-0-eu", "audit-1-eu", …

The occurrence counter is stored in the recurring task’s metadata["occurrence"] field in zart_tasks. You can query it directly for introspection:

SELECT metadata->>'occurrence' AS occurrence
FROM zart_tasks
WHERE task_id = '__zart_recurring__:nightly-report';

Waiting for an Occurrence to Complete

Use DurableScheduler::wait to block until a specific occurrence finishes:

let record = durable
    .wait("report-3", Duration::from_secs(30), None)
    .await?;

println!("Status: {:?}", record.status);
println!("Result: {:?}", record.result);

Stopping a Recurring Task

To cancel a recurring task entirely, use DurableScheduler::stop_recurring:

let durable = DurableScheduler::from_backend(&pg);
durable.stop_recurring("nightly-report").await?;

This marks the recurring task as inactive. No new occurrences will be scheduled, but executions that are already running will complete normally.

Full Example

For a complete, runnable example demonstrating all three overlap policies (SkipIfRunning, CancelAndRestart, AlwaysStart), see the recurring-durable example in the repository.

The example shows three scenarios:

Scenario	Overlap Policy	What You’ll See
Inventory Snapshot	`SkipIfRunning`	Second tick skipped while first runs
Config Refresh	`CancelAndRestart`	Stale run cancelled, fresh one starts
Audit Window	`AlwaysStart`	Multiple runs overlap in parallel