Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

ADR-0032: Durable Workflow Runtime

Status

Accepted

Date

2026-08-22

Owners

Mayfly runtime maintainers

Context

Mayfly currently provides the execution kernel used by Cylon: cooperative coroutines, ready queues, virtual timers, waits, cancellation primitives, and static dependency graphs. Cylon builds higher-level persona sequences directly with Scheduler::spawn_with_deps, channels, and application-owned completion logic.

That division worked for a linear review demonstration but fails for a durable autonomous workflow:

  • scheduler completion does not distinguish successful work from a failed, blocked, cancelled, or changes-requested outcome;
  • a completed dependency can release downstream work even when its outcome should have blocked the path;
  • retry, correction, external approval, and timeout behavior is application code rather than a reusable Mayfly contract;
  • closures and in-memory channels cannot be serialized or reconstructed after restart;
  • the Mayfly WAL and PAL contracts are not yet connected to a replayable workflow state model;
  • each consumer can invent incompatible state, retry, and recovery semantics.

Mayfly needs a workflow layer above the scheduler. The scheduler must remain a small application-neutral execution kernel; it must not learn about Cylon personas, source-control providers, LLMs, QA commands, or product lifecycle states.

Decision

Mayfly will add a durable workflow runtime whose source of truth is a versioned, append-only event history interpreted by a deterministic reducer. Applications submit serializable workflow specifications and register handlers for stable handler names. The runtime decides which node attempt is eligible, records that decision durably, and then asks the scheduler to execute the handler.

The key distinction is:

Workflow state is durable policy state. Scheduler state is a transient execution effect used to advance that workflow state.

Layering and dependency direction

flowchart TB
    App["Application policy, for example Cylon"]
    Workflow["Mayfly workflow crate"]
    Reducer["Pure specification validator and reducer"]
    Runtime["Runtime effects and handler registry"]
    Scheduler["Mayfly scheduler crate"]
    Journal["WorkflowJournal trait"]
    Wal["Mayfly WAL reference implementation"]
    Store["Application journal implementation"]

    App --> Workflow
    Workflow --> Reducer
    Workflow --> Runtime
    Runtime --> Scheduler
    Runtime --> Journal
    Wal -. implements .-> Journal
    Store -. implements .-> Journal
    App --> Store

The crate rules are:

  1. scheduler remains unaware of workflows and application domains.
  2. A new workflow crate owns specifications, commands, events, state, reduction, handler contracts, and the scheduler effect adapter.
  3. The reducer module performs no I/O and does not read wall-clock time, environment variables, global randomness, or scheduler state.
  4. WorkflowJournal is defined beside the workflow contract. The existing wal crate provides a local reference implementation and depends on the contract, never the reverse.
  5. Applications may supply another journal implementation. Cylon can therefore use its Postgres-backed WAL without making Postgres a Mayfly core dependency.

Public identity contract

Every durable identity is an opaque, strongly typed value in the public Rust API and a canonical string in serialized data.

IdentityScope and stability
WorkflowIdNames a logical workflow definition across compatible revisions.
WorkflowVersionImmutable version selected when a run is created.
WorkflowRunIdGlobally unique execution identity; never reused.
NodeIdStable within one workflow definition and preserved across compatible revisions.
NodeRunIdIdentifies one node’s execution within a workflow run.
AttemptIdGlobally unique attempt identity. Retries always receive a new value.
EventIdGlobally unique idempotency key for one journal fact.
SignalIdGlobally unique delivery identity for an external signal.
ArtifactIdOpaque reference to governed output stored outside bounded events.

Runtime-generated global identities use UUIDv7 by default so operators can sort them approximately by creation time. Generation is injected behind an IdSource in deterministic tests. No correctness rule depends on UUID time ordering; journal sequence is authoritative.

NodeId is author-defined, human-readable, and restricted to a stable ASCII identifier grammar. Renaming a node is a compatibility change because event history and dependency edges reference it.

Serializable workflow specification

A run pins an immutable specification containing at least:

  • workflow ID, workflow version, schema version, and content hash;
  • node IDs, stable handler name and handler contract version;
  • bounded serialized input or artifact references;
  • dependency and transition declarations;
  • retry, deadline, cancellation, and budget policies;
  • output names made available to downstream nodes.

Rust closures are never serialized. An application registers an implementation for (handler_name, handler_version) when constructing the runtime. Resume is blocked if a handler required by an unfinished run is unavailable or incompatible.

The initial durable representation is UTF-8 JSON with explicit schema_version fields. The Rust API is format-neutral so another encoding can be added later without changing state semantics.

Commands, events, state, and effects

The runtime follows a decide-persist-reduce-effect loop:

  1. A command is validated against the current state.
  2. decide(state, command) returns versioned events or a typed rejection.
  3. The journal atomically appends the events using the expected next sequence.
  4. reduce(state, event) derives the new state.
  5. Newly committed state may request an external effect such as dispatching a handler, scheduling a timer, or cancelling an active task.
  6. Effect results return as new commands and repeat the loop.

The reducer is total over every supported event version: it either returns a new state or a typed corruption/compatibility error. It never ignores an unknown event.

Representative commands include:

  • create or resume a run;
  • start or complete a node attempt;
  • deliver an external signal;
  • advance a virtual deadline;
  • request cancellation;
  • acquire or renew recovery ownership.

Representative facts include:

  • run created;
  • node became ready;
  • attempt dispatched, completed, timed out, or cancelled;
  • retry scheduled;
  • signal wait opened, accepted, rejected, or expired;
  • node outcome committed;
  • workflow completed, failed, blocked, or cancelled;
  • recovery ownership acquired or released.

Exact Rust type names and payload fields land in Stories 1.2 and 1.3. This ADR fixes their semantics and dependency direction.

State model

stateDiagram-v2
    [*] --> Pending
    Pending --> Ready: dependencies and conditions satisfied
    Pending --> Skipped: branch not selected
    Pending --> Cancelled: cancellation propagated
    Ready --> Running: attempt dispatch committed
    Ready --> Cancelled: cancellation requested
    Running --> Waiting: durable signal or timer wait
    Running --> Succeeded: successful outcome committed
    Running --> RetryPending: retryable outcome
    Running --> ChangesRequested: corrective outcome
    Running --> Failed: permanent failure or budget exhausted
    Running --> Blocked: missing handler or external prerequisite
    Running --> Cancelled: cancellation acknowledged
    Waiting --> Running: signal accepted
    Waiting --> RetryPending: timeout is retryable
    Waiting --> Failed: terminal timeout
    Waiting --> Cancelled: cancellation propagated
    RetryPending --> Ready: backoff deadline reached
    ChangesRequested --> Ready: correction edge starts new attempt
    Succeeded --> [*]
    Failed --> [*]
    Blocked --> [*]
    Cancelled --> [*]
    Skipped --> [*]

ChangesRequested is a non-success outcome used by corrective workflows. A transition may route it back to the same logical node or to another node, but the next execution always has a new AttemptId and consumes configured budgets.

Typed outcomes and dependency policies

At minimum, node outcomes distinguish:

  • Succeeded;
  • ChangesRequested;
  • RetryableFailure;
  • PermanentFailure;
  • Blocked;
  • Cancelled;
  • Skipped.

The safe default dependency policy is all_succeeded. Merely reaching a terminal state does not imply success. Other policies such as all_completed and any_succeeded must be explicit in the specification.

Outputs carried in events are bounded. Story 1.3 sets the following contract:

  • inline output is compact JSON limited to 64 KiB per outcome;
  • non-success reasons are limited to 4 KiB of UTF-8;
  • an outcome may carry at most 32 evidence references;
  • large, sensitive, or independently governed output is represented by an ArtifactId, byte length, lowercase SHA-256 digest, and optional media type.

all_succeeded releases a node only after every direct prerequisite succeeds. Once every prerequisite is terminal and that condition is impossible, a cancelled prerequisite propagates cancellation, a blocked prerequisite propagates blocking, and other non-success combinations skip the dependent. all_completed releases after every direct prerequisite has any terminal outcome. any_succeeded releases immediately after its first successful prerequisite; if every prerequisite finishes without success, the same cancel/block/skip propagation applies.

Independent ready or running branches continue after another branch fails. The workflow terminal result is derived only after every node has a terminal outcome: permanent failure takes precedence over blocked, which takes precedence over cancelled; a graph containing only success and explicit skip outcomes succeeds. ChangesRequested remains a non-terminal control outcome that may select a Story 1.4 correction edge. RetryableFailure follows the bounded Story 1.5 retry policy; when policy disallows recovery, an explicit termination fact makes the node fail while retaining the original outcome.

Story 1.3 emits event schema v2 for typed outcomes and stricter terminal-run facts. The reducer continues to accept schema-v1 histories produced by Story 1.2, including their immediate failure and blocked terminal facts.

Story 1.4 emits event schema v3 and fixes the conditional-transition grammar. An edge may match any outcome, one NodeOutcomeKind, or one named top-level JSON output field equal to a scalar value. Predicates cannot execute code, call handlers, read ambient state, or inspect clocks. Parallel edges fan out in specification order. Exclusive edges must be provably disjoint by outcome class and field/value constraints; potentially overlapping pairs fail validation.

Each activation records the stable edge position and source outcome event sequence. This fingerprint prevents replay or a later correction cycle from reusing a stale transition result. Conditional edges may form explicit correction loops; unconditional dependency edges remain acyclic. A graph with conditional cycles declares its initial nodes explicitly.

Reusable child definitions live in the root specification’s closed, versioned catalog. References are validated before run creation, recursion is rejected, and nesting is limited to eight child levels. Child start and completion events carry the parent node and globally unique child run ID. Input and output mapping copies only declared top-level fields. Every child call selects either propagate or require_success, so child failure and cancellation cannot be silently interpreted as parent success.

Story 1.5 emits event schema v4 and run-state schema v2. RetryPolicy pins a maximum attempt count, eligible outcome classes, immediate/fixed/exponential backoff, deterministic optional jitter, and an optional per-attempt deadline. The safe default permits three attempts, retries only RetryableFailure, and leaves ChangesRequested to explicit correction transitions. Retrying PermanentFailure requires the specification to name that class explicitly.

VirtualTime is an unsigned millisecond offset from the run epoch. The pure runtime accepts only monotonic AdvanceTime commands; the Story 1.9 scheduler adapter will translate Mayfly clock deadlines into those commands. A fixed backoff adds its declared delay. Exponential backoff uses min(initial * multiplier^(next_attempt - 2), maximum). Optional jitter adds a value in 0..=maximum_ms from a fixed FNV-1a calculation over the pinned seed, run ID, node ID, and next attempt number. The computed absolute deadline is stored in RetryScheduled, so replay never recomputes or consumes an attempt.

RetryReleased is accepted only after the recorded deadline. Immediate retry still emits both schedule and release events at the current virtual time. Handler invocation receives the one-based attempt number, unique AttemptId, deterministic run/node/attempt idempotency key, and optional absolute deadline. An attempt identity can be consumed only once within a run.

The default run ceilings are 10,000 attempts and 100,000 transitions, with no elapsed limit unless the application pins one. Initial readiness, conditional activation, retry scheduling, and retry release consume transition budget; handler and child starts consume attempt budget. Exhaustion records its kind, immutable limit, observed usage, and most recently committed outcome. Node attempt exhaustion fails that node. Run attempt, transition, or elapsed-time exhaustion immediately fails the run, after which every command is rejected.

Delivery guarantee: at-least-once invocation, committed-once outcome

Mayfly cannot promise exactly-once execution of arbitrary application side effects. A process may crash after a handler changes an external system but before Mayfly records the returned outcome.

The public guarantee is therefore:

  • a handler attempt may be invoked more than once during recovery;
  • every invocation receives the same AttemptId and idempotency key until that attempt has a committed terminal outcome;
  • the journal commits at most one terminal outcome for an attempt;
  • repeating the same event is idempotent;
  • a conflicting second terminal outcome is rejected as corruption;
  • once committed, a successful attempt is never normally dispatched again;
  • handlers performing external effects must use the attempt idempotency key or reconcile their effect before returning.

This is at-least-once handler invocation with committed-once outcomes.

Journal and snapshot guarantees

Each run has a monotonically increasing journal sequence. Append uses compare and-swap semantics against the expected next sequence and deduplicates by EventId. A durable backend must not acknowledge append until its declared durability policy is met.

Snapshots are derived caches containing:

  • workflow run state;
  • last included journal sequence, event ID, and complete record checksum;
  • checksums of the pinned specification and complete derived state;
  • an explicit snapshot schema version.

Snapshots are never the source of truth. A stale, corrupt, or missing snapshot is discarded and rebuilt from events. Journal gaps, conflicting sequences, or unknown event versions fail closed.

Story 1.6 defines WorkflowJournal in the storage-neutral workflow crate and implements LocalWorkflowJournal in wal. commit_command enforces the decide-persist-reduce boundary and validates that a backend acknowledges the exact records expected before exposing new state. Append batches use expected-next-sequence compare-and-append semantics; exact replays are idempotent and conflicting event IDs or sequences fail closed.

The local format stores version-1 MWJ1 journal frames and MWS1 snapshot frames as bounded JSON with CRC-32 corruption checks. A whole per-run journal file is replaced atomically through a same-directory temporary file. The selected durability mode controls whether acknowledgement follows write, userspace flush, or file and directory synchronization. Operations are safe against concurrent callers in one process. Cross-process ownership remains a Story 1.7 recovery-lease responsibility rather than an implicit filesystem locking guarantee.

Normal execution

sequenceDiagram
    participant App as Application
    participant RT as Workflow Runtime
    participant J as Journal
    participant S as Scheduler
    participant H as Handler

    App->>RT: create run with pinned WorkflowSpec
    RT->>J: append RunCreated and NodeReady
    J-->>RT: committed sequence
    RT->>RT: reduce committed events
    RT->>J: append AttemptDispatched(attempt_id)
    J-->>RT: committed sequence
    RT->>S: schedule registered handler
    S->>H: invoke(input, dependencies, attempt_id)
    H-->>S: Succeeded(output)
    S->>RT: complete attempt command
    RT->>J: append AttemptCompleted and NodeSucceeded
    J-->>RT: committed sequence
    RT->>RT: reduce and unlock success dependents

The attempt dispatch fact is committed before invocation. The success fact is committed before dependents observe success.

Story 1.9 implements this boundary with HandlerRegistry, WorkflowRuntime, and SchedulerAdapter. A ready handler contract is resolved before StartNode is committed. Its request contains cloned durable input, dependency outcomes, attempt metadata, signal resolution, and a read-only May scheduler cancellation handle, but no mutable workflow state or journal. Independent ready attempts are all durably started before their May coroutines run. Returned outcomes are accepted only while their exact AttemptId remains active; late or cross-attempt results fail closed.

Handler panics and scheduler termination without a result become bounded typed attempt failures. Durable cancellation targets map to the exact transient task correlation. May clock wakes return authoritative VirtualTime values which the runtime persists through AdvanceTime; the reducer still reads no clock. Recovery reconciliation invokes the original attempt ID and idempotency key, while restored retry deadlines become May scheduler timer effects.

The application owns scheduler nesting. A coordinator that is not itself a May task uses WorkflowRuntime::execute and the standalone SchedulerAdapter. A coordinator already running inside a May task uses execute_in_current_task and execute_recovery_in_current_task; it must not construct and drive a second scheduler while occupying the caller task. The current-task mode executes ready handlers serially in specification order and inherits the caller’s cancellation handle and coroutine stack. It rejects attempt deadlines because enforcing a concurrent timeout requires the standalone adapter. This split preserves one May scheduler ownership boundary without introducing another executor or reactor.

Inspection and operational telemetry

Story 1.10 defines schema-versioned inspection and PAL projections above the durable state. RunInspection contains ordered node and attempt summaries, blocking diagnostics, deadlines, attempt counts, and typed outcome classes, but never handler input or inline output bodies. Bounded non-success reasons are the operator-safe diagnostic channel. Diagnostic history is capped and reports complete, retained, and omitted counts.

PAL facts are a payload-free projection of acknowledged journal records plus explicit recovery operations. PAL is never fed back into reduction and may be lost or sampled without affecting replay. A deterministic causal key joins run, node, and attempt scope across handler spans and PAL; durable projections also retain journal event ID and sequence.

Metrics have a closed type system rather than string labels. The supported dimensions are run state, node state, failure class, outcome class, recovery mode, and fixed latency bucket. Workflow, node, run, attempt, and handler IDs cannot become metric labels. The metrics crate publishes complete projections atomically for exporters.

Retry and correction

sequenceDiagram
    participant RT as Workflow Runtime
    participant J as Journal
    participant C as Virtual Clock
    participant H as Handler

    RT->>H: invoke attempt 1
    H-->>RT: RetryableFailure(reason)
    RT->>J: append AttemptCompleted and RetryScheduled(deadline)
    J-->>RT: committed
    RT->>C: register virtual deadline
    C-->>RT: deadline reached
    RT->>J: append NodeReady and AttemptDispatched(attempt 2)
    RT->>H: invoke attempt 2
    H-->>RT: Succeeded(output)
    RT->>J: append AttemptCompleted and NodeSucceeded

Retries, corrections, and transitions each consume declared budgets. A handler cannot reset attempt counts or extend a run budget. Backoff uses Mayfly’s clock abstraction and is deterministic under virtual time.

The committed Story 1.5 sequence uses NodeOutcomeCommitted, RetryScheduled, and RetryReleased; attempt start is represented by NodeStarted. Deadline expiry records VirtualTimeAdvanced followed by AttemptDeadlineExpired. The scheduler registers the absolute deadline only after the journal accepts the scheduling fact.

Durable external wait

sequenceDiagram
    participant H as Handler
    participant RT as Workflow Runtime
    participant J as Journal
    participant X as External Actor

    H-->>RT: wait for signal approval with deadline
    RT->>J: append SignalWaitOpened(token_hash, deadline)
    J-->>RT: committed
    Note over RT: process may stop and recover here
    X->>RT: deliver Signal(signal_id, token, decision)
    RT->>J: append SignalAccepted(signal_id, decision)
    J-->>RT: committed
    RT->>H: resume with accepted signal

Signals are scoped to a run and waiting node, carry a unique SignalId, and present the expected opaque token. Duplicate accepted signals are idempotent; stale, mismatched, or conflicting signals cannot advance the run.

Story 1.8 implements this contract in event schema v5. Raw tokens are non-serializable and debug-redacted; only their canonical SHA-256 digest enters SignalWaitOpened and derived state. SignalAccepted and SignalRejected consume the delivery identity exactly once, while SignalExpired records the wait deadline after the authoritative VirtualTimeAdvanced fact. A duplicate identity returns a typed already-consumed disposition without another state transition. Recovery restores the original attempt, name, digest, and deadline.

Structured cancellation

sequenceDiagram
    participant A as Requester
    participant RT as Workflow Runtime
    participant J as Journal
    participant S as Scheduler
    participant H as Active Handler

    A->>RT: cancel run(reason, requester)
    RT->>J: append request, descendant outcomes, RunCancelled
    J-->>RT: committed
    RT->>S: cancel recorded active targets
    S->>H: cooperative cancellation signal

Cancellation is not failure and never authorizes success. Active handlers see a cancellation handle; pending descendants receive recorded terminal outcomes according to the specification’s propagation policy.

Story 1.8’s initial policy is explicit cascade. One atomic cancellation decision records requester, bounded reason, current virtual timestamp, policy, and active handler/child targets, then records a cancelled outcome for every non-terminal local descendant and the terminal run fact. Effects occur only after that durable acknowledgement: the Story 1.9 adapter will map recorded targets to scheduler cancellation and child-run propagation. The scheduler substrate already exposes the same atomic cancellation observation through TaskContext::is_cancelled and reports a distinct Cancelled completion.

Restart recovery

sequenceDiagram
    participant R as Recovering Runtime
    participant J as Journal
    participant P as Recovery Lease
    participant HR as Handler Registry
    participant S as Scheduler

    R->>P: acquire run ownership lease
    P-->>R: ownership granted
    R->>J: load latest valid snapshot and event tail
    J-->>R: snapshot plus ordered events
    R->>R: validate versions, hash, sequence, and reduce
    R->>HR: verify unfinished handler contracts
    HR-->>R: compatible
    alt committed terminal attempt
        R->>R: do not dispatch again
    else dispatched without terminal outcome
        R->>S: reconcile or re-invoke same AttemptId
    else pending retry or signal
        R->>S: restore timer or external wait
    end

Only one runtime may own recovery for a run. Story 1.7 defines the storage-neutral RecoveryLeaseStore/RecoveryLeaseGuard contract and retains the guard in ResumeSession. The local WAL backend implements non-blocking, cross-process ownership with an OS file lock; distributed stores can provide their own equivalent fencing policy behind the same contract.

The public recovery modes are deliberately distinct:

  • inspect_run reconstructs read-only state from a validated snapshot and journal tail when available;
  • replay_run ignores snapshots and reduces complete authoritative history;
  • resume_run acquires ownership, reconstructs, checks pinned specification and unfinished-handler compatibility, then returns ordered recovery effects.

An invalid snapshot is disposable and triggers complete-journal fallback with a diagnostic. Invalid journal history or incompatible contracts fail closed. Completed nodes produce no effects. A started handler without a terminal fact produces ReconcileAttempt carrying the original attempt identity and idempotency context; it is never silently marked successful or assigned a new attempt. Durable waits, retry deadlines, and active child runs likewise produce explicit restoration or reconciliation effects.

Version and compatibility policy

Workflow durability begins only with explicit compatibility rules:

  1. Every specification, event, snapshot, and handler contract has its own version field.
  2. A run pins the exact workflow version and specification content hash.
  3. Additive optional fields require deterministic defaults and may remain in the same schema version.
  4. Removing, renaming, or changing the meaning of a field requires a new schema version and a tested upcaster from supported older versions.
  5. Event type meaning is immutable after release. Corrections use a new event type or version; historical facts are never rewritten.
  6. Unknown event or snapshot versions fail closed. They are never skipped.
  7. Handler compatibility is explicit. An unfinished node resumes only with an implementation declaring compatibility with its pinned handler contract.
  8. Node IDs referenced by active histories cannot be silently renamed.
  9. Golden serialized fixtures and replay tests are compatibility gates in CI.
  10. The initially supported persistence format is development-stage until the first tagged Mayfly release containing workflow; breaking changes before that tag still require fixtures and migration notes.

Application boundary: Cylon

Cylon maps its provider-neutral Workpiece to a versioned WorkflowSpec and stores the resulting WorkflowRunId with the Workpiece. It registers handlers for persona turns, workspace operations, review, QA, and external approval. Because Cylon’s daemon coordinator is already a May scheduler task, it invokes the workflow through the current-task execution and recovery entry points; nesting a standalone Mayfly scheduler in that task is incompatible with this ownership boundary.

Mayfly sees only handler names, serialized inputs, typed outcomes, artifacts, and policy. It does not know that a handler represents Hera, Tyrol, Leoben, or Athena; that QA uses just nt; or that an artifact is a Git diff.

Cylon remains responsible for rules such as:

  • only Athena can authorize coding Workpiece completion;
  • Leoben can request changes or block but cannot complete;
  • QA evidence is bound to an exact workspace root, base revision, and diff hash;
  • any edit after QA invalidates that evidence;
  • external issue trackers and pull requests are optional projections.

This boundary allows other Mayfly consumers to build durable workflows without depending on Cylon concepts.

Resolved decisions

  • Workflow orchestration belongs in Mayfly, not in each Cylon executor task.
  • The scheduler remains the execution kernel and does not own durable workflow policy.
  • State is event-derived; snapshots are disposable acceleration structures.
  • Specifications use registered handler names, not serialized closures.
  • The initial durable encoding is versioned JSON.
  • Handler invocation is at least once; terminal attempt outcome is committed once.
  • Success-aware dependencies fail closed by default.
  • Time and identifier sources are injected for deterministic tests.
  • Journal storage is pluggable; Postgres is not required by Mayfly core.

Follow-up decisions by story

TopicOwning story
Exact Rust types, validation errors, and serialized golden fixtures1.2
Outcome payload limits, artifacts, and dependency policy API1.3
Declarative predicate grammar and child-workflow mapping1.4
Retry formulas, jitter seeding, and run budget accounting1.5
Local journal framing, checksums, fsync, and snapshot cadence1.6
Recovery lease backend contract and interrupted-attempt reconciliation1.7
Signal token lifecycle and cancellation propagation API1.8
Scheduler adapter and handler registry API1.9
PAL names, inspection DTOs, metrics, and redaction1.10
Cylon workflow specification and migration flag1.11

These are implementation details within the boundaries decided here, not unresolved ownership questions.

Alternatives considered

Keep workflow logic in Cylon

Rejected. Every consumer would recreate retry, failure propagation, external wait, recovery, and audit semantics. Cylon’s current manual persona chain is the motivating example of this duplication.

Expand Scheduler directly with workflow concepts

Rejected. It would mix transient coroutine mechanics with durable policy, couple low-level scheduler users to serialization, and make the kernel harder to test and reuse.

Serialize coroutine stacks and closures

Rejected. Stack snapshots are runtime- and build-specific, cannot provide a stable compatibility contract, and capture ambient state that should instead be explicit workflow input.

Promise exactly-once handler execution

Rejected. It is impossible for arbitrary external side effects without a transaction shared by Mayfly and every external system. Stable attempt idempotency keys and committed-once outcomes provide an honest recoverable contract.

Make a database the workflow API

Rejected. Mayfly defines behavior and durability traits; storage technology is an implementation choice. A mandatory database would harm deterministic tests, embedded use, and local recovery.

Use PAL as the durable source of truth

Rejected. PAL is an operational stream and may be sampled or dropped. Workflow events require ordered durable append and replay guarantees.

Consequences

Positive

  • Cylon can express correction loops and completion policy without owning a bespoke workflow engine.
  • Failures, cancellation, changes requested, and success become mechanically distinct.
  • Recovery and audit behavior becomes consistent across Mayfly consumers.
  • Pure reduction and virtual time enable exhaustive deterministic tests.
  • Storage and application integrations remain replaceable.

Costs and risks

  • The runtime introduces new stable serialized contracts that require migration discipline.
  • Handlers must be idempotent or reconcile external effects after ambiguous crashes.
  • Event history, snapshots, artifacts, and leases add operational complexity.
  • A workflow can still be logically incorrect; Mayfly guarantees faithful execution of declared policy, not that application policy is wise.
  • The Cylon migration must temporarily support both the manual chain and the Mayfly workflow path until live parity is proven.

Security and safety

  • Journal events do not contain credentials or unrestricted model/tool output.
  • External signals require scoped opaque tokens and idempotent delivery IDs.
  • Handler registrations are explicit; workflow data cannot select an arbitrary executable or shell command.
  • Cancellation, timeout, and budget decisions are durable and cannot be reset by handler output.
  • Unknown versions, conflicting outcomes, journal gaps, and invalid snapshots fail closed.

Implementation plan

Implementation follows Epic 001 Stories 1.2 through 1.11. Story 1.2 begins with the pure serializable core. No Cylon migration begins until the Mayfly contracts it consumes have deterministic tests and compatibility fixtures.