Postmortem: May Runtime Substrate Violation

Incident status

• Date discovered: 2026-08-23
• Affected repositories: Mayfly and its May integration
• Severity: Architectural integrity failure
• Status: Remediated; dependency ratchet and substrate boundary remain enforced
• Related decision: ADR-0033: May Runtime Ownership and Dependency Compatibility

Executive summary

Mayfly introduced MIO and Tokio-based execution paths without first examining the May runtime that Mayfly is built upon. This created competing runtime ownership inside one process.

May already owns coroutine scheduling, work stealing, timers, parking, cancellation, native I/O readiness, and coroutine resumption. On Unix it uses epoll/eventfd or kqueue directly; on Windows it uses IOCP. Its generator dependency supplies architecture-specific assembly for stackful context switching. Mayfly nevertheless added a second MIO poller and a second runnable queue. The daemon also constructed an independent Tokio runtime for Hyper.

The result is not a harmless implementation alternative. It is a split-brain runtime: Mayfly records task policy as if it controls coroutine execution while May continues to schedule the actual coroutine. A may::coroutine::yield_now() returns the coroutine to May’s scheduler; the Mayfly scheduler retains only a JoinHandle and cannot resume that coroutine itself.

The incident was caused by a failure to understand and document the execution substrate before selecting familiar ecosystem libraries. Review and CI checked whether code compiled and tests passed, but did not test whether a dependency claimed ownership already held by May.

What happened

The relevant changes accumulated through path dependence rather than a new execution requirement:

1. Commit ce12be2 created a PyOS-style generator scheduler before May was a Mayfly dependency. It owned ready, sleep, and wait maps because no execution substrate had yet been selected.
2. Commit 099cc81 inserted May beneath that existing design but retained Mayfly’s runnable queue and JoinHandle model. No design review reassigned execution ownership to May.
3. Commit 5097c52 added optional MIO polling under the async-io feature on 2025-06-30.
4. Commit 889e773 expanded that feature into a separate MIO event loop on 2025-07-01.
5. The July 2025 overhaul carried the hybrid scheduler forward wholesale.
6. The daemon HTTP implementation added Tokio and Hyper, later retaining reqwest as a test dependency and notify, which also resolves MIO.

There was no demonstrated requirement for another execution scheduler. The second scheduler was an inherited abstraction that survived the May adoption, then accumulated features because its original ownership assumptions were not re-examined.

The optional feature made the architectural fork appear isolated. In reality, it duplicated an execution concern and allowed the two scheduler paths to diverge. A later spawn-request change exposed one symptom: the MIO loop could exit before draining a request when it began with no tasks. Fixing that loop condition would address only the symptom, not the invalid ownership model.

Impact

At discovery:

• Mayfly had two scheduler loops with different idle, spawn, timer, and I/O behavior.
• The Mayfly ready queue did not own the coroutine it purported to make ready.
• Completion, cancellation, and readiness can be observed by one runtime while execution is controlled by another.
• Feature-specific tests can pass while exercising semantics unlike the production coroutine substrate.
• Tokio and its reactor add a third execution domain in the daemon.
• Mayfly was locked to crates.io may 0.3.51, so the local May repository’s runtime work and fixes are not consumed or validated by Mayfly.
• Future workflow recovery work risks encoding durable guarantees on top of an execution adapter whose lifecycle semantics are unsound.

No data-loss event has been attributed to this incident. The primary impact is loss of architectural correctness and confidence. Because the defect affects ownership and lifecycle boundaries, absence of an observed data-loss event is not evidence of safety.

Evidence from May

Inspection of the local May repository found an existing, complete substrate:

• src/scheduler.rs creates worker queues, work stealers, timers, selectors, and coroutine pools.
• src/io/event_loop.rs binds the native selector loop to the May scheduler.
• src/io/sys/unix/epoll.rs uses epoll and eventfd directly and schedules the exact parked coroutine on readiness.
• src/io/sys/unix/kqueue.rs provides the corresponding kqueue implementation.
• src/io/sys/windows/iocp.rs handles overlapped I/O, cancellation, timeouts, and coroutine resumption through IOCP.
• src/io/sys/unix/co_io.rs and wait_io.rs expose coroutine-aware raw file descriptor I/O.
• src/park.rs, src/sleep.rs, and src/yield_now.rs park and resume the concrete CoroutineImpl.
• generator uses architecture-specific assembly to preserve registers and swap stackful execution contexts.

May’s dependency graph contains neither Tokio nor MIO.

Dependency audit

The 2026-08-23 all-platform Cargo metadata audit found these direct edges into forbidden runtimes:

Every package with direct or transitive reachability to these runtimes is marked as a versioned incompatibility in .config/runtime-dependency-policy.toml. The ten direct edges produce 18 package/runtime reachability records. That file is a removal ledger, not an approval list.

Root cause

The root cause was an incorrect architectural assumption:

May was treated as a coroutine primitive that Mayfly needed to schedule, rather than as the runtime that already schedules, parks, wakes, and performs I/O for those coroutines.

This led directly to several invalid secondary assumptions:

Contributing factors

• ADR-0003 said the design was coroutine-first and decoupled from Tokio, but it did not name May as the exclusive execution owner.
• No dependency policy rejected async runtimes or reactor crates.
• Dependency review considered direct manifest entries but not reverse, transitive, dev, feature, and target-specific paths.
• The MIO feature was not exercised consistently by CI, allowing its loop to drift from the default implementation.
• There were no substrate contract tests proving which runtime parks and resumes a coroutine.
• Work in the local May fork was not pinned into Mayfly, making it easy to reason from an incomplete external version instead of the maintained code.
• Proposed May design documents were not clearly separated from implemented facilities during earlier reasoning. The implemented native I/O and scheduling code is sufficient evidence; proposals must not be presented as landed behavior.

What went well

• The architectural conflict became reproducible through scheduler behavior rather than remaining a latent dependency concern.
• Source-level inspection of May exposed the actual ownership boundary.
• Cargo metadata produced a complete, versioned inventory of current Tokio and MIO ingress paths.
• The durable workflow layer remains sufficiently separated that its pure reducer and journal work need not be discarded.

Where detection failed

• Code review asked whether the new poller worked, not whether Mayfly had the authority to own a poller.
• CI validated formatting, compilation, linting, and behavior but had no architecture check.
• Documentation used generic terms such as “coroutine scheduler” and “async I/O” without assigning exclusive component ownership.
• Dependency additions did not require an execution-model compatibility assessment.

Corrective actions

Containment completed

• Record the incident and the invalid assumptions.
• Accept ADR-0033 assigning exclusive runtime ownership to May.
• Inventory direct and transitive Tokio/MIO paths across Cargo metadata.
• Add a CI dependency ratchet that rejects new or changed direct and transitive forbidden-runtime paths.

Remediation completed

• Select and pin a tested May revision rather than the unrelated crates.io release resolution.
• Delete Mayfly’s MIO dependency, async-io feature, IoSource, poll state, token registry, and duplicate scheduler loop.
• Replace scheduler coroutine handoff with a May coroutine-aware dispatch gate; May owns the concrete park, wake, and cancellation operations.
• Refactor Mayfly so it owns workflow/task policy but never claims to resume a May coroutine.
• Replace daemon Tokio/Hyper HTTP with a May-native implementation.
• Remove unused notify hot-reload scaffolding; any future implementation requires a May-native design.
• Replace the Reqwest test client and remove all Tokio-dependent test utilities.
• Remove each temporary policy entry as its edge disappears; the final policy state contains no exceptions.
• Exercise policy dispatch, join, virtual workflow timers, cancellation, and May-native HTTP I/O in the scheduler and daemon test suites.
• Extend substrate contract coverage to raw descriptor adapters and durable recovery when those execution-adapter APIs are added.

Prevention

Every new or updated dependency must now answer these questions:

1. Does it construct, require, or assume an async executor?
2. Does it own an I/O reactor, timer wheel, signal loop, or task lifecycle?
3. Does any feature, dev dependency, build dependency, or target-specific dependency introduce Tokio or MIO?
4. Can it operate on May CoIo/WaitIo primitives without blocking a May worker or moving lifecycle ownership elsewhere?
5. Has Cargo metadata been audited with all features and all targets?

A “blocking” facade over a Tokio-backed library is still incompatible. Moving an incompatible runtime to another thread is still incompatible. Hiding it behind an optional Cargo feature is still incompatible.

The automated audit is a minimum guard, not proof of compatibility. Libraries using another executor, hidden global threads, blocking system calls, or a private reactor require the same architectural review even when they do not depend on packages named tokio or mio.

Lesson

Runtime architecture must be discovered from the substrate’s source before an abstraction is selected. In Mayfly, convenience at the dependency boundary cannot override single ownership of coroutine execution.