Runtime

What every compiled Hale binary always ships with. Always- loaded; not optional; no import needed; the substrate every Hale program depends on.

Reference material, synced from the compiler repo’s spec/. The guide is the gentler path in.

What every compiled Hale binary always ships with. Always- loaded; not optional; no import needed; the substrate every Hale program depends on.

This document distinguishes the runtime (always there) from the standard library (stdlib.md, importable but bundled). Go’s distinction between runtime and other stdlib packages is the model: runtime is automatic; stdlib is explicit.

Naming note: The language is Hale; the runtime/ substrate concept is called lotus, and the C-runtime symbols stay lotus_* (per project memory). When this doc says “lotus” it means the substrate; “Hale” means the language proper.

What’s in the runtime

Section titled “What’s in the runtime”

Memory

Section titled “Memory”

Lifecycle

Section titled “Lifecycle”

Scheduler — multi-scheduler cooperative

Section titled “Scheduler — multi-scheduler cooperative”

Lotus uses a multi-scheduler cooperative model (closest existing analog: Erlang BEAM, not Go’s M:N). The reasons are framework-discipline:

Concurrency comes from running multiple cooperative schedulers in parallel (one per CPU core, by default). Loci belong to a specific scheduler; cross-scheduler communication uses the bus just like cross-process communication. Loci may be migrated between schedulers transparently for load balancing because all their communication is bus-mediated already.

Specifically:

Placement classes (per-locus execution strategy)

Section titled “Placement classes (per-locus execution strategy)”

Just as projection class governs a locus’s memory strategy, placement class governs its execution strategy. Placement is a deployment seam, not an intrinsic property of the locus (see spec/decisions.md § F.31). Placement entries live in a placement { } block on main locus only, parallel to bindings { } for bus topology:

main locus App {
params {
gateway_a: Gateway = Gateway { venue: "venue-a" };
gateway_b: Gateway = Gateway { venue: "venue-b" };
metrics: MetricsServer = MetricsServer { port: 9100 };
ui: Renderer = Renderer { };
}
placement {
gateway_a: pinned(core = 1);
gateway_b: pinned(core = 2);
metrics: cooperative(pool = io);
ui: cooperative(pool = render);
// unspecified main-locus params → cooperative(pool = main)
}
}

Placement remains honestly bimodal: either a locus shares a cooperative pool (an OS thread running a cooperative drain loop) or it owns its own OS thread. There is no third position.

Class Yield discipline Resource
cooperative(pool = X) (default for unspecified main-locus params, with X = main) Yields between substrate cells (handler exit, lifecycle transition, bus dispatch, time::sleep, explicit yield). time::sleep slices into ≤100ms intervals and folds in lotus_bus_queue_drain for the locus’s pool after each slice, so cells posted by other threads deliver mid-loop even during a long keep-alive sleep. Handler bodies are atomic. Shares pool X’s OS thread with other cooperative loci placed on the same pool.
pinned / pinned(core = N) / pinned(cores = A..B | A..=B | {a, b, c}) / pinned(node = N) / pinned(l3 = name) / pinned(..., replicas = K) No yield to siblings; owns its OS thread. Bus events to/from cross-thread boundaries via formal mailbox post. Dedicated OS thread. core = N pins it to one CPU; cores = ... (Phase 1a) sets its mask to a core set; node = N / l3 = name (Phase 1b) set the mask to a NUMA node / cache domain from topology { } (and bind the arena there); replicas = K (Phase 1c) fans into K single-threaded instances, one per core. The OS schedules freely within the mask. Linux-only; best-effort no-op elsewhere.

Pool inference rule. The cooperative pool set is inferred from cooperative(pool = X) references in the placement { } block. The runtime spawns one OS worker thread per inferred pool name beyond main (which is always the program’s main thread). No separate threads { } declaration block at v1 — when per-pool attributes (priority, affinity, realtime hint) become useful, the block lands then as a typed extension. Pool main exists in every program regardless of whether placement { } references it.

Nested-instantiation inheritance. Placement entries apply only to top-level main locus params fields. Loci instantiated nested in another locus’s body (in birth / run / lifecycle methods, or as let-bound children) inherit the parent’s pool. There is no way to spell “this nested child runs on a different pool than its parent” — that would require the nested-instantiation expression to carry placement, which would re-mix the deployment and intrinsic layers F.31 separates.

Topology block (Phase 1b). A main locus may also declare a topology { } block — a declare-only description of the host’s core partition, a sibling deployment seam to placement { } / bindings { }:

topology {
reserve cores 0..2; // held back for the OS / main
node 0 {
l3 fast { cores 4..8; } // a CCD / shared-L3 group
l3 slow { cores 8..12; }
}
node 1 {
l3 heavy { cores 12..16; }
}
}

A placement { } entry then targets a domain: pinned(node = 0) sets the thread’s affinity mask to node 0’s core set (the union of its L3 domains — here {4..12}), and pinned(l3 = fast) sets it to the named domain’s cores ({4..8}). The compiler resolves the domain to a concrete core set at compile time (closed-world: node ids and domain cores are literals), reusing the same cpuset affinity mechanism as pinned(cores = ...). Validation (unique node ids, globally-unique L3 names, non-overlapping domains, no domain/reserved overlap, and every pinned(node/l3) referencing a declared domain) is static. L3 domain names go through the identifier rule, so a hard keyword (e.g. bulk) can’t name a domain.

Replicas (Phase 1c). pinned(..., replicas = K) is the parallelism sugar: it fans the field into K single-threaded instances, replica i pinned to one core of the affinity set (round-robin — pinned(cores = 4..12, replicas = 8) puts replica i on core 4 + i; more replicas than cores wraps; with no affinity the K instances are OS-scheduled). This is deliberately not a multi-worker pool — a cooperative pool is one consumer thread, and the lock-free rings, bus devirtualization, and single-threaded-method guarantee all rest on that invariant. Parallelism instead comes from more single-threaded units, each its own single consumer, so every invariant survives. replicas is pinned-only (K cooperative loci on one pool would share a thread, which isn’t parallel) and composes with the topology targets (pinned(node = 0, replicas = 4) fans across node 0’s cores with each replica’s arena bound to node 0). Codegen emits K instantiations at the field’s init site; all K register their bus subscriptions (a subscribed topic fans out to every replica) and all K are joined + dissolved at parent teardown via the deferred- dissolve frame. The replicas are non-addressable — there is no field[i] surface; they are workers that pull from the bus or run their own loop.

The dispatch plan (GH #476 Change 8). Which lowering a subject’s dispatch receives — dynamic (runtime hash lookup), static_bucket (compile-time subject id, dispatch still queued), static_direct (synchronous direct calls to every subscriber) — is a CONCLUSION, not a fact declared anywhere in the program. A program has one plan, derived once (DispatchPlan::from_gates) from one gate set: the bus graph’s sites, each keyed by its wire subject, and the bundled stdlib’s sites after them (the logger’s log.** and its sinks, which every program’s lowering dispatches). Lowering lowers that plan, the execution identity frames its digest, and the canonical model holds it restricted to the subjects the program’s own bus sites name and the loci the program declares. The bus graph’s per-subject eligibility gates decide the flavor (a single ladder, DispatchFlavor::of, which the backend calls rather than re-deciding; static_direct takes three legs: every publisher and subscriber same-thread, every handler quiet, and the gate’s payload_flat column, a payload struct whose every field is an inline scalar — a direct-eligible subject with a managed payload is static_bucket), and the Change-8 arrangement supplies each row’s publisher/subscriber thread domains and same_domain — “every publish site and every subscriber of this subject sit in one domain”, the precondition the future placement-driven flavors (GH #464) need, withheld rather than guessed whenever the model cannot place a locus’s WHOLE population — a locus with both an arranged instance and a dynamic birth is incomplete, and its arranged instance does not answer for the instances the model admits it cannot see. A locus’s domains are found by the gates’ own spelling of it, the canonical post-merge name, never the display name, so a locus of an imported seed has its domains like any other (before F.40 phase 3’s C5 they were found by the display name, which demangles an imported locus and never met the gates’ spelling, and every such row printed no domain). Plan subjects are WIRE subjects; hale model dump prints the model’s restriction of the plan and its same-domain count.

The gates’ placement leg reads the placement table (F.40 phase 3, P1): a type is same-thread only when every instance of it runs on main, so static_direct needs every publisher and every subscriber type to be. An instance nested under a root field placed off main runs on that field’s thread, an adapter in bindings { } on its own, and an instance the table cannot place (a literal in a scope whose domain is unknown) is never taken for main; each keeps its subject on a queued flavor. The same answer decides where a subscriber’s bounded(N, …) is legal (main-queue registrations only, spec/decisions.md F.37).

Binding roles and replica indices in the model. A binding’s role is the authored role: kwarg when present and otherwise the inferred one — publish-only is connect, subscribe-only is listen, and an ambiguous or unused binding is a typecheck diagnostic, never a default. One rule serves both the desugar and the canonical model (hale_syntax::desugar::binding_role_for); before GH #476 Change 8 the desugar ran its half AFTER topic references were rewritten to literal subjects, so the publish and subscribe ends were always empty and no role was ever inferred — every binding had to spell role: or codegen refused it. Loss behavior follows the role, because the runtime does: the connect side is the publish side, where a send failure marks the entry lost and or wait parks through the reconnect window; the listen side re-arms on peer EOF and a link it cannot serve is structural. A pinned(..., replicas = K) field contributes K instances whose model rows carry their own 0-based INDEX — the same i codegen bakes and a keyed subscriber registers under — and the model refuses a replica set that is not contiguous from 0. Replica-ness belongs to the LAST path component: the arrangement walks into each replica, so what a replica owns (App.workers[0].leaf) is an ordinary child that inherits its owner’s domain and carries no index of its own.

A subject named by a bindings { } entry is never devirtualized: an external peer is (or may be) the real counterparty, so the dispatch has to go through the transport the binding realizes. That exemption is keyed at BOTH grains — the topic decl name a binding entry writes and the wire subject a desugared program dispatches on — because the backend builds its graph after topic desugaring, and a decl-name-only gate silently let a bound subject be bucketed past its own adapter.

The plan is part of what a build is: its digest is framed into the executable identity (exec_digest) alongside the toolchain hash, the compiler version, the build options, and every source byte. The plan framed there is the one the backend lowers: derived once from the bus graph over the merged, topic-desugared program the backend walks, with the same ladder and the same domains as the model’s plan (one domain map, from the arrangement the model’s rows are projected from; F.40 phase 3, C5). The plan’s digest covers what lowering reads, each subject’s flavor and subscribers; the same_domain column is consulted by no flavor today, so the byte it held is reserved and always 0, until the same-domain flavors (GH #464) lower by it. Two builds of identical sources that lower dispatch differently — notably the all-dynamic LOTUS_NO_BUS_DEVIRT=1 control arm — therefore have different identities, and a recording made under one is refused against the other rather than replayed against a bus that behaves differently.

Thread + memory co-location. A pinned(node = N) / pinned(l3 = name) locus binds not just its thread but its memory: its arena is created via lotus_arena_create_labeled_on_node, which flags the arena’s NUMA node, and every chunk that arena grows is mmap’d (page-aligned) and bound to the node with the mbind syscall (MPOL_BIND) before first touch — so pages fault in on the node regardless of which thread touches them first (the locus struct is instantiated on main but runs on its own pinned thread). Sub-regions inherit the node, so a node-pinned locus’s method scratch — the dominant per-invocation allocation — lands on its node too. mbind is invoked as a raw syscall, so this adds no libnuma dependency; the whole feature is zero-cost for programs that don’t opt in (an unbound arena, the default, takes the ordinary malloc / chunk-pool path byte-for-byte). Best-effort and Linux-only, exactly like pinned(core = N): an mbind the box can’t honor (node absent, capability denied) falls back to first-touch, and on non-Linux hosts the arena allocates normally. (Huge-page-backed chunks and node binding don’t currently combine — a node-bound arena uses regular pages; a follow-up.)

Single-threaded-method invariant. A locus’s methods may be invoked only on the OS thread that owns its placement’s pool. Cross-pool method calls and lateral field accesses go through the bus’s existing copy-and-condvar dispatch machinery, which already crosses thread boundaries safely. The typechecker walks the static call graph from each top-level placement entry, propagates pool ownership through method receivers, and rejects calls that cross pools without going through the bus. This is the substrate enforcement that makes M:N safe — without it, multi-pool deployments would silently race on locus arenas (which are unsynchronized bump allocators).

The single shared bus payload arena is the deliberate exception to “one owning thread per arena”: it is reachable concurrently from any pool, because some stdlib primitives allocate their result there directly rather than into a per-locus arena (e.g. std::io::tls::recv_bytes, which always targets it regardless of the caller’s per-thread scratch). That arena therefore carries an internal lock on its bump — two pinned loci calling such a primitive at once do not corrupt each other’s allocations. Per-locus arenas keep the lock-free bump; only this one shared arena pays for the lock.

Why no “greedy” class

Section titled “Why no “greedy” class”

A natural temptation is to want a third option: “shares a pool’s thread but doesn’t yield.” That would be a bimodality violation. Cooperative already guarantees handler-level atomicity — no preemption within a substrate cell — so the only thing such a class could add over cooperative is “don’t yield between cells either.” But that means leaving the shared pool entirely. The place you go when you leave is your own thread. That’s pinned.

Latency-critical work, or anything that genuinely shouldn’t share with neighbors on its pool, is signaling that it belongs on its own pool (placed cooperative(pool = some_quiet_pool)) or on its own thread (placed pinned). The first option is new in F.31: pools partition the cooperative substrate so “shouldn’t share with siblings” no longer forces pinned — but the underlying bimodality (cooperative-pool vs pinned-thread) holds.

time::sleep drain semantics

Section titled “time::sleep drain semantics”

The codegen lowering of std::time::sleep(d) slices the request into intervals of at most 100ms and ends each slice’s EINTR-retry loop with an inline call to lotus_bus_queue_drain against the program-wide cooperative queue (plus a pinned-mailbox drain). The total wall-clock sleep is preserved (the slices sum to d); sleeps of 100ms or less take exactly one slice, so the common case is unchanged. A cooperative subscriber looping

run() {
while !self.bail {
std::time::sleep(100ms);
// ... loop body sees handlers fired during the sleep
}
}

receives cells posted by other threads — unix-bound reader threads, pinned publishers via lotus_bus_local_dispatch, etc. — right when each sleep returns, without an explicit yield;. The drain is idempotent, so existing code with sleep; yield; stays correct.

Which path delivers to whom. The queue drained here is the program-wide in-process cooperative queue, drained on the main thread — so this sleep-loop drain delivers in-process cells to a cooperative subscriber on the main pool. It is not the only delivery path: cross-process topics (udp:// / unix:// bound via LOTUS_BUS_CONFIG) are delivered by the transport reader thread, which dispatches directly into the subscribed handler set and reaches a cooperative locus on any pool — provided that pool’s thread is free to run the dispatch. So a non-main cooperative subscriber receives reliably as long as it doesn’t monopolize its pool thread with a blocking call. (Pinned loci are a third path: a per-locus mailbox drained at each sleep/yield.) The failure mode is a cooperative subscriber whose run() blocks — the dispatch can’t run and its handlers never fire; that blocking-and-subscribing combination is what the dead-bus-receiver rule rejects (see “Type-check rules” in spec/semantics.md), not non-main placement on its own.

Long sleeps no longer starve main-pool handlers. Before the slicing, the drain happened only after the whole sleep returned, so the natural keep-alive idiom

run() { while true { std::time::sleep(60s); } } // on the main thread

starved every pool = main bus handler for 60s at a time: a main-pool subscriber registers with coop_pool == NULL, so the wire/reader-thread dispatch path lands its cells on the global cooperative queue (g_bus_queue_for_remote, the same object only main drains). A 60s blocking sleep on main left those cells unserviced for the full 60s — indistinguishable from “the handler never fires.” Slicing keeps the queue serviced ~10×/s during any sleep, so a main-pool handler that republishes onto a topic an async-pool subscriber listens to (e.g. a udp-reader-fed producer forwarding to per-connection writers) now flows promptly regardless of how long main is asleep.

The pinned-mailbox path is unchanged: pinned subscribers wake on lotus_mailbox_post’s condvar broadcast regardless of what the cooperative scheduler is doing.

Owner-executed handlers (2026-07-15, downstream handoff)

Section titled “Owner-executed handlers (2026-07-15, downstream handoff)”

Two runtime rules restore the single-threaded-locus invariant (F.31) dynamically — the compiler already enforces it for direct calls, but two bus paths used to violate it under load (reproduced as a SIGSEGV in a 10k msg/s ingest bench):

  1. The global cooperative queue is drained only by its owner thread (main, recorded at queue creation). The scope-exit flush emitted at the end of every fn/method body — and the sleep-slice / yield drains — call lotus_bus_queue_drain on whatever thread ran the body; on any thread but the owner the call is now a no-op. Previously a pinned publisher’s flush would execute a main-pool subscriber’s handler on the publisher’s thread, concurrently with main’s own drains (the locked drain releases the queue mutex before each handler invocation) — two threads inside one locus.
  2. Payload deserialization happens on the subscriber’s owner thread. The non-flat dispatch paths deserialize each published payload into the subscriber’s arena (Task-11 arena routing); for a target owned by a different thread this write used to happen on the publisher’s thread — an unlocked cross-thread write into a foreign arena. Now a cross-thread publish enqueues the wire bytes plus the deserialize fn in the cell, and the owner materializes (deserializes into its own arena) at drain, just before invoking the handler. Same-thread targets take the same wire cell (2026-10-11): the deserialize-at-dispatch path they kept wrote the payload’s owned fields into the subscriber’s locus arena for good (spec/memory.md § Cross-thread wire cell per-delivery reclaim).

Consequences: a main-pool subscriber’s handlers run only on main (at sleep-slice / yield / scope-exit drains — worst-case ~100ms after a cross-thread publish, per the slicing above); pool subscribers run only on their pool’s worker; pinned subscribers only on their own thread. Flat (pointer-free POD) payloads are exempt from rule 2 — a verbatim byte copy writes no arena. Delivery and FIFO order per subscriber are unchanged.

Item B from the 2026-05-21 friction log (a cooperative publisher’s <- to a pinned subscriber) is resolved as of 2026-06-01 — the earlier “drains only at dissolve in some configurations” was a sequencing effect, not a lost wakeup. The mailbox condvar path is correct: a pinned subscriber whose run() returns proceeds into the blocking lotus_mailbox_drain_one, and a cooperative publisher’s <- wakes it via the not_empty broadcast — confirmed by coop_to_pinned_mid_program::pinned_returning_run_drains_mailbox.

The residual constraint is inherent to a single pinned thread: a pinned subscriber with a long-running run() that never returns or yields cannot drain its mailbox during run() — the one thread is busy in the loop. Such a run() must reach a cooperative yield point (time::sleep / yield) for the TLS-cached lotus_mailbox_drain_pending to service the mailbox (the coop_to_pinned_mid_program sleep-loop test), or let run() return so the post-run blocking drain takes over. This is a property of the bimodal model — “pinned owns its thread, no yielding between cells” — not a bug; a pinned subscriber that must receive bus traffic while busy should yield in its loop. Cross-binary flows (unix / shm_ring with their own reader threads) land in g_bus_queue and benefit from the sleep-folded drain above.

Long-running cooperative children: placement closes Item D

Section titled “Long-running cooperative children: placement closes Item D”

Pre-F.31, declaring a long-running cooperative child as a params field (metrics_server: std::http::Server = ...) serialized parent and child onto the main thread: the child’s run() body never returned, so the parent’s run() never started. The workaround was the sibling-in-main pattern — hoisting the child to a top-level main locus param so it ran “alongside” the parent rather than “inside” it.

Under F.31 the sibling-in-main pattern IS the canonical shape, and it composes cleanly because placement { } lets each sibling pick its own pool:

main locus App {
params {
gateway: Gateway = Gateway { };
metrics: std::http::Server = std::http::Server { port: 9100 };
}
placement {
gateway: pinned(core = 1);
metrics: cooperative(pool = io);
}
}

Parent’s run() no longer serializes against metrics’s accept loop — they sit on different OS threads. To shut down gracefully when one finishes (e.g. a duration-bounded gateway exits), call metrics.shutdown() from the finishing locus’s thread; the C-iii interruptible-accept work makes this the supported pattern.

The nested-as-child shape (long-running cooperative locus as a params field of a non-main locus) remains structurally serialized — that’s a consequence of the nested-instantiation inheritance rule (children share their parent’s pool by construction). Nested long-running children are an antipattern under F.31: hoist to main-locus siblings.

Typecheck enforcement. The compiler rejects the antipattern at typecheck. A non-main locus with a non-trivial run() body holding a params field of a locus type whose own run() is also non-trivial — including std::http::Server and the other entries on the known-long-running stdlib allowlist — gets a hard error pointing at the canonical sibling-in-main + placement fix. The runtime starvation that motivated this rule is silent (the parent’s run() simply never executes), so the type-side rejection is load-bearing: it converts a class of hard-to-diagnose runtime bugs into a clear compile-time signal.

The rule asks whether a run() is long-running, which is not whether it never returns, and the two are two definitions, two columns of each locus’s run row (hale_types::flows::RunRow, read through the flow rows):

Every body that never returns is long-running; the converse does not hold. A stdlib locus, whose body the checker does not see, is both when it is on the known-long-running allowlist.

where async_io — green-I/O cooperative pools (F.35)

Section titled “where async_io — green-I/O cooperative pools (F.35)”

The sibling-in-main fix puts each long-running child on its own OS thread, which caps concurrent connections at one-per-pool. To scale beyond that without spawning a thread per connection, a placement entry may declare where async_io:

main locus App {
params {
listener: std::websocket::Server = std::websocket::Server { ... };
worker: WsWorker = WsWorker { ... };
}
placement {
listener: cooperative(pool = ws_accept) where async_io;
worker: cooperative(pool = ws_workers) where async_io;
}
}

where async_io opts the pool into green-I/O scheduling: the pool’s worker drain loop integrates an epoll instance, and blocking I/O syscalls inside locus methods on this pool park- and-resume instead of blocking the OS thread. The user code inside the locus is unchanged — recv_bytes(stream) reads the same line of source whether the pool is async_io or not; the substrate picks the right lowering at the syscall boundary. std::http::Server is the stdlib’s case of it: placed on an async_io pool it serves each connection on a coroutine of its own, and anywhere else one connection at a time (spec/stdlib.md § “Server concurrency”); the TCP connection substrate it is built on places its listener and its consumers apart (pinned accept, async_io serving: spec/stdlib.md § “The TCP connection substrate”).

One worker thread per named cooperative pool — a promise, not an implementation detail (Crumb batch-3, 2026-07-28). Every named cooperative pool (async_io or classic) has exactly one OS worker thread for the lifetime of the program. Consequences a consumer may depend on: every run(), bus handler, and parked- coro resume for loci on the pool executes on that one thread (coroutines interleave on it; they never migrate), so a thread-affine C library — a JS engine, SQLite in serialized mode, a GUI toolkit — placed on a named pool is entered from a single thread by construction, and @export re-entry from foreign code called on that thread stays on it. Concurrency within a pool is cooperative only. If a scale-out mechanism ever lands, it will be a new opt-in placement form; cooperative(pool = name) keeps the single-worker guarantee.

Because parking yields the shared worker, N reader loci that each park on their own fd — the F.35 one-reader-per-signal shape — are serviced concurrently by a single pool. Two invariants make that multiplexing correct: (1) the drain loop starts a queued run() the moment the running coro parks, not only when it completes, so a long-lived reader never starves the readers queued behind it; and (2) each coro’s caller-arena — the thread-local that decides where its stdlib allocations (recv result blobs, string builders) land — is snapshotted across the park and restored on resume, so a coro that resumes after a sibling ran (and perhaps dissolved) never allocates through an arena the sibling has since torn down. Every blocking-recv primitive on the pool honors invariant (1) by parking rather than blocking recvfrom/read (the std::io::udp recv family joined the tcp/tls siblings here in the 2026-07-15 downstream handoff, and std::time::sleep joined them 2026-07-28 (Crumb batch-5) via a timer-only park — a deadline with no fd, serviced by the same expiry sweep — so sleeping coros overlap on one worker and a sleep inside a handler never blocks the pool; the timer half of an event loop now costs what the design always claimed).

Each bus delivery to a subscriber on an async_io pool runs its handler on a coroutine (a struct + a 64 KiB stack). Rather than allocate and free that pair per delivery, the pool keeps a bounded per-worker free-list (cap 64) of completed coro slots and reuses them — a warm fan-out skips the per-dispatch stack allocation entirely (2026-07-16). The free-list is worker-thread-local (no lock) and drained at pool teardown, so a busy async pool retains up to 64 × 64 KiB (~4 MiB) of coro stacks at steady state. Transparent to user code — a pure allocation optimization, no behavior change.

A handler’s payload is its own until it returns, across parks (GH #781, 2026-09-19). The storage a delivery’s payload lives in belongs to the coroutine that runs the handler, for the whole invocation: the cell’s inline payload bytes are copied into the coro when it starts, a spilled heap payload and the wire path’s per-delivery subregion transfer to it, and all three are released once the handler returns — however many parks later. So a handler that parks on a sleep, a socket read, or a subprocess drain and then reads its payload parameter again reads what it read at entry; concurrent deliveries to the same subscriber never share payload storage. This did not hold before: the drain dequeued each cell into a stack local and handed the handler a pointer into it, so a parked handler returned to a frame the next dequeue had already reused, and every parked delivery read the LAST published value (the same mechanism leaked a >512-byte spilled payload per park). Pinned subscribers were never affected — a mailbox cell outlives the handler it dispatches, which has no coro to park on.

Typecheck rules:

See spec/decisions.md § F.35 (forthcoming) for the green-I/O substrate design + perf-axis trade-offs.

(Compare: rich / chunked / recognition projection classes are genuinely three-way because N≈10, N≈30, and N≈300 are different cost regimes at scale — memory has more genuine intermediate ground than time does. Placement, even with M:N pool partitioning, stays bimodal.)

Cross-class bus semantics

Section titled “Cross-class bus semantics”

Implementation status (m26 + m27 + m28a + m28b + m28c; F.31 pending)

Section titled “Implementation status (m26 + m27 + m28a + m28b + m28c; F.31 pending)”

m25 wired the annotation through parse / typecheck / codegen. m26 ships cooperative semantics; m27 ships pinned threads (run-only); m28a lifts pinned to full lifecycle; m28b lights up cross-thread bus mailboxes — pinned loci can subscribe and publish, with cells routed across threads via per-locus mailboxes; m28c adds optional CPU-core affinity via pthread_setaffinity_np.

F.31 (2026-05-23, Phase 1-5 + Phase 4a shipped): the placement-at-main surface and M:N cooperative pools. The per-locus : schedule annotation is removed; the placement choice moves to a placement { } block on main locus. Cooperative subscribers can be partitioned across N pools (N OS threads, each running its own lotus_coop_pool_worker drain loop against its own per-pool ring buffer); cross-pool bus dispatch reuses the m28b condvar+memcpy machinery. A pool’s worker thread can take the same affinity forms as pinned — cooperative(pool = X, core/cores/node/l3 = …), applied right after the worker spawns (2026-08-12; entries naming one pool must agree, and the main pool takes none). The single-threaded-method invariant — a locus’s methods may be invoked only from its pool’s thread or via the bus — ships as a typecheck rule (Phase 5).

Phase 4 v1 limit (handler-only cross-pool delivery). The runtime ships pool-aware bus dispatch for now: a subscriber whose enclosing locus is placed on a non-main cooperative pool gets its handler invoked on that pool’s worker thread. Lifecycle methods (dissolve / accept) still run on the main thread for cooperative-pool loci — the codegen does NOT yet relocate them to the pool worker (run and a pool-placed root’s own birth are relocated, below). For state mutated only inside bus handlers this is enough to honor the single-threaded-method invariant (the handler is the only writer of locus state on the pool thread). State touched by both lifecycle bodies and handlers on a non-main pool is a Phase 4b concern; the typechecker doesn’t flag it today because lifecycle methods are not user-callable in the recv.method() shape that Phase 5 checks. Plan: Phase 4b moves lifecycle dispatch onto the pool worker via the same queue mechanism (post “run_init” / “drain_exit” cells at instantiation / scope-exit boundaries). The pool-placed field’s run() is posted to its worker (below), and its subtree initializes there (§ “m27 + m28a”, the pool side): everything nested under it is built, registered and born on the worker, and a nested cooperative child’s run() runs there inline. The placed field’s own birth(), with its birth-epoch closures and its birth_check, runs on the worker too, as one more job the instantiating thread posts after the field’s registrations and waits for, before its readiness (§ “Lifecycle obligations”, line 3). Its accept and dissolve() still run on the instantiating and the teardown thread.

Runtime pool inheritance for in-method-body instantiation (2026-05-29). A locus instantiated inside a method or bus-handler body that is itself executing on a pool worker inherits that pool at runtime. Codegen has no static placement name for such a locus (placement keys on main-locus params fields only), so the run-posting site and the subscription-registration site resolve the pool as: the compile-time-known pool when the locus IS a placed main-locus field, else the pool whose worker is currently on-CPU (lotus_coop_pool_current(), which reads the per-thread g_current_pool_tls; NULL on the main thread). When the resolved pool is non-NULL the child’s run() is posted to it (its own cell — and, on an async_io pool, its own parkable coro, so a blocking recv in the child’s run() parks that child’s coro rather than the spawning handler’s) and its bus subscriptions are tagged with that pool so dispatch routes to the right worker. The run() post is gated on the child being owned beyond the spawning scope — accept’d, an owned param field, or returned. A handler-local let-bound long-lived locus is not owned (its deferred dissolve fires at the handler’s scope exit), so posting its run() would execute after it’s dissolved; its run() stays synchronous. Its subscriptions are tagged with the pool all the same (§ “Subscriptions follow the tower”: a run-time subscriber takes the constructing thread’s route whatever its ownership). They used to go to the global queue, where main ran their handlers beside the worker’s own use of the locus. The canonical N-dynamic-children shape (per-connection handlers, per-tenant workers) is therefore an accept’d child whose run() holds the recv loop — it multiplexes on an async_io pool by construction. See spec/memory.md § “Owned param-field child allocation” for the companion arena rule that makes the owned child’s full subtree outlive the spawning frame.

Adapter loci instantiated inline in bindings { Topic: AdapterLocus { ... }; } are NOT main-locus params fields and so receive no placement { } entry. Their run() recv- loops need a dedicated thread by construction; the substrate places them pinned-equivalent implicitly (same m90 routing + pthread spawn that pre-F.31 fired via the adapter’s : schedule pinned annotation). The annotation goes away but the behavior is preserved automatically — the bindings-inline shape unambiguously signals “transport adapter with a recv loop.”

The implementation notes below describe pre-F.31 shapes (m25 through m28c). They remain accurate for the cooperative-only- on-main-pool case, which is the v1-compatible default when no placement { } block is declared.

m26 (cooperative): Each <- enqueues (handler, self, payload_copy) cells onto a program-wide FIFO queue (@lotus.bus_queue.global) instead of running handlers inline. The scheduler drain loop pops cells one at a time and invokes the handler — handler-atomic per substrate cell, with cooperative yields BETWEEN cells rather than nested call frames. Handlers may publish more events; drain continues until empty.

Drain runs at the start of every flush_dissolve_frame — before any long-lived locus dissolves — so subscribers process pending cells while still alive. Plus an explicit yield; statement (m26b) drains at user-placed points inside long internal loops. v0 limitation: cells enqueued DURING a dissolve are leaked.

The C runtime gained the queue surface:

ptr lotus_bus_queue_create(void)
void lotus_bus_queue_enqueue(ptr q, ptr handler, ptr self, ptr payload)
void lotus_bus_queue_drain(ptr q)
void lotus_bus_queue_destroy(ptr q)

m20’s “memcpy payload into subscriber’s arena” step happens at ENQUEUE time (publisher’s frame).

m27 + m28a (pinned threads + full lifecycle): Pinned-class loci spawn a pthread at instantiation; the locus’s params and its full declared lifecycle (birth → run → drain → dissolve, each only if declared) execute on that thread, in order.

A pinned locus’s subtree initializes on its thread. The instantiating thread creates the locus’s mailbox, if it has one, and then its thread. The thread makes the mailbox current and initializes the locus’s params: every nested construction, its subscriptions, its birth() and a cooperative child’s inline run(), and the params bracket and its settle. The initialization has a scope of its own: a temporary locus a default builds (Helper { }.value()) is dissolved when the initialization ends, on the pinned thread, not when the instantiating function’s scope does. So every lifecycle body of a locus nested under a pinned one runs on the pinned thread, the domain it is placed in, and a yield inside one (std::time::sleep, an await) drains the pinned locus’s mailbox, as a yield on main drains main’s queue. The instantiating thread waits until the params are initialized (lotus_pinned_start_await_ready), so nothing observes the tree before it is built. While it waits it services its own mailbox exactly as a yield on it would: on main it drains main’s queue, on a pinned thread that thread’s mailbox (a yield on a pool worker drains neither, and neither does this wait). So a nested body that waits during the initialization for a reply from a subscriber on the instantiating thread gets it. Then it finishes the instantiation (the locus’s own subscriptions; its synthetic fields and failure route were stored before the thread was created, so the initialization never reads one unset) and releases the thread (lotus_pinned_start_go) into birth() and the rest of its lifecycle, and continues. An override written at the literal (Worker { started: pthread_self() }) is the instantiating code’s: it is evaluated on the instantiating thread, before the pinned thread starts. A locus an override builds as the field’s value is part of the subtree, and is built on the pinned thread. At scope exit (deferred- dissolve flush), pthread_join blocks until the pinned thread has finished its lifecycle and returned; the main thread’s only remaining work for a pinned entry is the join plus the locus’s arena destroy wholesale (drain / dissolve are SKIPPED on the main side — they ran on the pinned thread).

m28a synthesizes a per-locus __pinned_main_<LocusName> function whose signature matches pthread’s start-routine contract directly (ptr (ptr)); pthread_create gets that function pointer with the locus’s start block as its argument: the locus, the start gate, and each value of the instantiating function that the params’ initialization reads (the instantiating thread runs nothing but its own mailbox’s handlers while they are read, and those reach their own subscribers, not its frame). No C-side adapter. The synthesized body makes the mailbox current, runs the params’ initialization (__pinned_init_<LocusName>), reports ready, waits for its release, calls each declared lifecycle method in sequence, then returns null.

The pool side. A root field placed on cooperative(pool = X) initializes its subtree on X’s worker the same way, as the first job of that field. Its params’ initialization is lowered into __pool_init_<LocusName>, the start block is the pinned one, and the instantiating thread posts the initialization to X as one job (lotus_pool_start_post) and waits for it as for a pinned locus (lotus_pool_start_await_ready, which services its own mailbox the same way). The worker runs every nested construction, its subscriptions (routed to X), its birth() and a cooperative child’s inline run(), and the params bracket and its settle, so a failure a nested child raises during the initialization is delivered on the worker at settle. The instantiating thread then finishes the instantiation (the field’s own subscriptions; its synthetic fields were stored before the post), posts the field’s own birth step (its birth(), birth-epoch closures and birth_check, lowered into __pool_birth_<LocusName>) to X as a second job and waits for it the same way (§ “Lifecycle obligations”, line 3), and posts the field’s run() to X behind both, as before. The three are X’s jobs in that order: init, birth, run. An override written at the literal is evaluated on the instantiating thread before the post, as for a pinned locus. A delivery to the subtree during the initialization is X’s own: it runs on the worker, at a yield of the initialization or after it. A yield inside the initialization (std::time::sleep, yield;) drains X’s queue on the worker, as a yield on a pinned thread drains its mailbox; a yield inside a cell that drain runs drains nothing, and outside an initialization a yield on a pool worker drains nothing, as before. The job never parks: on an async_io pool it runs on the worker’s own stack, not a coroutine, so a sleep or a socket wait inside it blocks the worker, and the initialization is complete before the worker starts another cell. The roots of one pool therefore initialize in post order, each complete before the next is posted. Ordinarily the job waits behind cells already queued on X: a field placed after a sibling on the same classic pool whose run() never returns is never initialized, and the instantiation waits for it (the sibling’s run() already holds the worker against everything else on X). No wait may be on itself (§ “Lifecycle obligations”, line 1): when the worker is itself blocked on the instantiating thread, waiting for the decision on a held failure that thread gives only at settle, it runs the posted initialization in place, still on the worker, and waits on. The constructor offers this pending initialization independently of queue capacity and tries to enqueue without blocking. Once the worker claims the pending initialization, the constructor proceeds to the readiness wait even if the queue remains full; it need not enqueue a second copy. The pending slot and any queued copy each hold a reference, and only one path runs the initialization. A target without threads initializes on the instantiating thread.

m28b stage 1 (inline-payload queue): Bus queue cells now carry an inline [u8; 512] payload buffer (with pthread_mutex_t guarding the cell array) instead of a pointer to subscriber-arena memory. The publisher memcpy’s into the cell at enqueue; the drain (running on the subscriber’s thread) memcpy’s from the cell into the subscriber’s arena before invoking the handler. This makes the queue the single point of cross-thread synchronization: each per-locus arena stays single-threaded territory, the boundary between layers is where the lock lives. Per spec/memory.md, “every locus boundary copies the payload” still holds — just with two memcpy’s per cell instead of one.

m28b stage 2 (cross-thread mailboxes): Each pinned locus that declares bus subscribe, or nests one that does, allocates its own lotus_mailbox_t at instantiation: a bounded ring buffer with pthread_mutex_t + pthread_cond_t + a shutdown flag, sharing the same inline-payload cell shape as the global queue. The locus’s struct grows a __mailbox: ptr field to hold it.

The bus entry table grows from {subject, self, handler} to {subject, self, handler, mailbox}. Cooperative subscribers register with mailbox = NULL; pinned subscribers register with their mailbox pointer. At dispatch time, the bus_dispatch fn loads entry.mailbox and branches: null → enqueue on the global cooperative queue (handler runs on the cooperative thread); non-null → lotus_mailbox_post on the pinned subscriber’s mailbox (handler runs on the pinned thread).

Subscriptions follow the tower. A locus nested under a root field placed off main runs on that field’s thread (§ Placement classes), and so do its bus handlers: its subscriptions register with its anchor’s route, not the global queue. That holds for a locus built at run time too — in a hook or a method, whatever owns it (a field it is stored in, an accepted child, or nothing beyond the constructing scope: a let or a statement-position literal the scope dissolves): on a pinned thread it registers with that thread’s mailbox, which the thread makes current before its params and its birth() run; on a pool’s worker, with that pool; on main, with the global queue. (std::http::Server builds its consumer in birth(); pinned, its deliveries used to reach the global queue and be served by main. A scope-local subscriber built on a pool’s worker or a pinned thread did the same until the registration stopped asking about ownership: main ran its handlers while the thread that built it was still using it.) A scope-local subscriber’s handlers run where that thread runs cells while the scope is open: on an async_io pool, while the constructing body parks; on a pinned thread, at the body’s sleeps and yields, which drain its mailbox; on main, at the drains of its statement boundaries and its scope exit. A classic pool’s cells are handler-atomic, and the constructing body holds the worker for the subscriber’s whole life, so a delivery to a scope-local subscriber there waits and is dropped with it; hale check warns at such a literal (spec/verification.md, “Dropped scope-local delivery”). Its dissolve is a reclaim like any other: the subscriber is quarantined before its storage goes, so a cell still queued for it on the pool or the mailbox is dropped at materialization (GH #703), and the release waits for a handler of it that has started (R52). A release on an async_io pool parks while it waits, and the reclaim state a thread keeps as stacks — releases entered, owner scopes, records deferred to a handler’s end — is the parked coroutine’s own: a park saves it and puts back the drain’s, so two coroutines dissolving their subscribers on one worker never pop each other’s entries. A pinned anchor has a mailbox if and only if something may post to it, whether or not it subscribes itself: a locus nested in its params tree subscribes, or a subscriber may be built at run time on its thread — a literal in one of its hooks or methods, or in its nested loci’s, or in any fn they reach through the call graph (free fns included, and the bodies of every locus those bodies build, a locus a params default builds included), of a locus that subscribes or whose construction template builds one (its params defaults, transitively, which are lowered wherever it is built), as the placement table’s dynamic sites record it; a literal whose enclosing scope’s domains are unknown (a fn passed as a value, which may run anywhere) counts as one that may run on every pinned thread. Any other pinned thread that subscribes to nothing has none, and its lifecycle is as before: its drain() and dissolve() follow its run() on its thread. A thread with a mailbox serves it after its run() returns, until the deferred-dissolve flush shuts it down (below): its drain() and dissolve() then run at main’s teardown, not as its run() returns. Under a pinned anchor that is the anchor’s mailbox; under an anchor on cooperative(pool = X) it is pool X. The route exists before the subscriptions that use it: the anchor’s mailbox is created before its params are initialized, which is where every nested instance registers. A pinned anchor’s params initialize on its own thread (m27 + m28a, above), so a nested instance that waits during its initialization for a delivery through the mailbox is served by the thread the mailbox belongs to. An anchor on a pool initializes its params on the pool’s worker (the pool side, above), so a nested instance’s bodies and its handlers run on the one worker, and a delivery during its initialization runs there at a yield or after it, never beside it. The route outlives the subscriptions: each nested instance deregisters in its own dissolve(), on the anchor’s thread, before the join below returns, and the join retires any registration still routed to the mailbox before destroying it.

The synthesized __pinned_main_<Locus> body grows a mailbox loop between run() and drain(): it calls lotus_mailbox_drain_one, which blocks on the condvar until either a cell arrives (returns 1, after dispatching the handler) or shutdown is signaled with empty queue (returns 0, breaking the loop). Pending cells flush before the loop returns 0 even after shutdown — the order check is “queue empty AND shutdown.”

Coordinated shutdown: at the deferred-dissolve flush, the main thread calls lotus_mailbox_shutdown on the pinned locus’s mailbox (sets the flag + broadcasts the condvar), then pthread_join. The pinned thread observes the empty+shutdown condition, breaks its loop, runs drain() and dissolve(), and exits — main joins, then destroys the mailbox and the arena.

Per The Design / lotus, this is the canonical “any → pinned” bus path: publisher and subscriber sit in different layers of the lotus, the substrate cost lives at the layer boundary (the mailbox lock + the inline payload’s two memcpy’s), and each arena stays single-threaded territory. Bimodality holds.

Still gated: pinned loci cannot declare accept() (children of pinned would need cross-thread cascade-dissolve coordination, which is meaningful new infrastructure beyond m28b’s mailbox post-and-continue) or a closure whose epoch is birth or dissolve — dissolve being the default with no epoch clause — (cross-thread routing inside the cascade). Tick, duration, explicit and inline closures fire on the pinned thread and are supported. The typechecker refuses the two gated shapes at the placement entry or adapter binding (adapters also run on their own pinned thread; rule 6); codegen keeps a backstop for builds that skip the checker.

m28c (CPU-core affinity): When a pinned locus declares : schedule pinned(core = N), codegen emits a call to lotus_set_core_affinity(tid, N) immediately after pthread_create succeeds. The C-side helper wraps pthread_setaffinity_np (with a cpu_set_t zeroed and bit N set) so codegen doesn’t have to know the cpu_set_t layout (opaque + size-variable across glibc versions). Best-effort semantics: if the requested core is unavailable (e.g., CI box with fewer cores than the source declares) or the syscall is denied, the runtime silently falls back to ordinary OS scheduling rather than refusing to run the binary. The underlying bimodality is unchanged — pinned(core = N) is a refinement WITHIN the pinned mode, not a third position.

Topology Phase 1a (cpuset affinity): pinned(cores = A..B) / pinned(cores = A..=B) / pinned(cores = {a, b, c}) generalize the single core to a core set: the thread’s affinity mask is the whole set and the OS schedules it freely within it — a range carves out an isolation domain rather than picking one CPU. Bounds are integer literals (placement is a closed-world deployment seam), so the compiler expands the spec statically — sorted, deduplicated — into a constant array and emits one call to lotus_set_core_affinity_set(tid, cores, count) after pthread_create. Range inclusivity follows expression ranges: .. excludes the upper bound, ..= includes it. The typechecker rejects a spec that selects no cores (4..4, 8..=4) and a duplicated set element; whether the cores exist on the deploy box stays best-effort at runtime — the C helper skips out-of-range indices and applies the mask only if at least one valid core remains. CPU affinity is Linux-only: on other hosts (macOS) both helpers are compiled as no-ops and the loci run unpinned. pinned(core = N) continues to route through the single-core helper unchanged.

Linker dependency: clang invocation now passes -lpthread unconditionally; small fixed cost in the resulting binary (libpthread is on every modern Linux).

Bus message router

Section titled “Bus message router”

The runtime’s bus is transport-agnostic. From the framework’s perspective, a transport is the bus kernel projected through a parameter regime: NATS and UDP multicast and TCP and Unix sockets are the same primitive (typed pub-sub) at different (B, c, σ, φ) values. The runtime knows about subjects, channels, and modes; specific transports come from stdlib (std::bus::*).

v1.x source surface. Subjects are now declared as typed top-level topic Foo { payload: T; subject: "..."; } decls (with optional : Parent for hierarchical wire subjects); deployment-time bindings live in the main locus’s bindings { Topic: <transport>; } block — the program’s OWN entry main: a main locus that arrives through import (a test seed importing the application it tests) keeps its bindings { } inert, so a test binary never binds, or unlinks, the sockets of the running application (GH #529 D7). It is also not counted against the one-main-locus rule, so a program that imports such a seed may declare a main locus of its own (GH #1059). Two transport shapes ship: substrate-provided unix("/path", role: ...) and user-supplied adapter loci named directly on the right-hand side (any locus satisfying __StdBusAdapter — fn send(subject: String, bytes: Bytes) — qualifies). In-memory delivery is absence-of-entry. Adapter bindings let protocol-layer transports (NATS, MQTT, TCP-with-framing, custom JSON-over-WebSocket) live in user code without the language having to enumerate protocol variants. See spec/semantics.md “Topic declarations → Phase 2” for the full surface, including the closed-world topology optimization that elides bus dispatch for unambiguous intra-locus and single-hop parent→child tower patterns when no binding is declared.

Binding realization failure (GH #227) + transports-as-loci (GH #233 steps 1–2, 2026-07-22). Source-level bindings { T: unix(...) } entries are sugar: codegen instantiates a stdlib transport locus (__StdBusUnixListenTransport / __StdBusUnixConnectTransport, runtime/stdlib/bus.hl) as a cooperative child at the main prelude — converging with the adapter path. The locus is the control plane; the data plane stays in C:

Publish fanout is untouched — realized entries land in the same g_bus_remote_entries table the fanout walks.

Connection-loss supervision (GH #233 steps 3–4). Publish fanout marks a locus-served connect entry lost on send failure (skipping it thereafter — down-window publishes drop, never falsely succeed) and pushes it onto a mutex’d pending list (lotus_bus_transport_mark_lost). The top of lotus_bus_queue_drain — owner thread, the only place failure handlers may run — drains the list (lotus_bus_drain_lost_transports): each handle goes to the codegen-registered dispatcher (lotus_bus_set_loss_handler, emitted only when main declares the matching on_failure) or straight to the structural exit (lotus_bus_transport_lost_fallback). The dispatcher invokes main.on_failure(main_self, transport_self, link_lost_violation) and, when the handler bumped __restart_count via restart (t), calls lotus_bus_transport_reconnect (re-runs connect-with-retry against the entry’s stored path; success clears lost). Codegen support: lotus.main.self global (stored at main-locus instantiation), lotus_bus_transport_bind_self (entry ↔ locus self, emitted after each connect instantiation).

Env-configured routes (LOTUS_BUS_CONFIG) have no source-level declaration to hang a locus on and keep the direct C path: lotus_bus_register_remote(subject, url, role) returns an i32 status (0 ok / -1 unrealizable — scheme/addr validation, socket/bind/listen, connect-retry timeout, thread spawn), with listener-side work done synchronously at registration (udp: parse + bind + multicast join via lotus_bus_udp_listener_setup). On failure the entry is popped (no dead slot for fanout to silently skip) and lotus_bus_load_config routes into lotus_bus_binding_fail. Config-route unix listeners share lotus_bus_unix_serve, so they re-arm identically. Normative contract: spec/semantics.md, “The publish contract”.

Adapter dispatch. At codegen, an adapter binding instantiates the adapter locus into the program-lifetime payload arena (same m90 routing the -> LocusRef(L) return path uses), resolves the locus’s send method’s fn pointer, and registers the (self, send_fn) pair with the runtime via lotus_bus_register_remote_adapter. The runtime stores both in lotus_bus_remote_entry_t’s adapter slot. Outbound fanout packages the wire bytes as an Hale-level Bytes value (built via lotus_bytes_from_buf against the lazy global payload arena) and indirect-calls send_fn(self, subject, bytes). No vtable lookup is needed at the runtime layer — codegen resolved the method at binding-emit time.

Adapter inbound (m105). Adapters receiving wire-bytes from their protocol layer call std::bus::__local_dispatch( subject, bytes); the primitive backs onto lotus_bus_dispatch_wire, which looks up the subject’s registered deserialize fn in the registration table (g_bus_table, the same table the publish-side fanout consults), reconstructs the struct-layout bytes, and fans into local subscribers via lotus_bus_local_dispatch. Symmetric to the unix reader- thread path; out-of-band recv loops (any code holding wire bytes for a bound subject) can use this too.

SPSC observation ring (GH #244). A single-producer fixed-slot ring over CALLER-PROVIDED memory, exposed as lotus primitives (lotus_spsc_init / _emit / _note_drop / _set_tag_b / _read) and the raw all-Int Hale surface std::ring::__spsc_*, every address an Int: statements __spsc_init(desc, data_off, tag_a, tag_b), __spsc_emit(seg_base, desc, ring_slots, w0, w1), __spsc_note_drop(desc) and __spsc_set_tag_b(desc, v), and __spsc_read(seg_base, desc, ring_slots, cursor_io, overruns_io, out, max) -> Int, the records copied. Built for observation planes (the iris observer attaches to these rings inside an shm segment, read-only, from a foreign process), so the layout is a STABLE documented contract:

Consumer cursors and overrun counters live OUTSIDE the shared segment (caller-owned); external readers never write the ring. This ring is the convergence target for iris’s observation protocol (its PROTOCOL.md pre-freeze sketch is this layout with tag_a/tag_b as sched_id/current_locus).

SHM ring substrate (Form K5). POSIX shared- memory ring backing the zero-copy bus route. Six C primitives in runtime/lotus_shm_ring.c, linked unconditionally so user programs that bind a topic to a zero_copy route resolve cleanly:

Layout in SHM: a 64-byte cache-aligned header (lotus_shm_ring_header_t — magic, slot_size, slot_count, atomic seqno) followed by N slots of slot_size bytes each. Header magic + sizes are validated on attach to catch ABI mismatches across binaries pinned to the same ring name.

Foreign-layout consumer (Proposal B). Two more primitives read an externally-defined ring described by a ring_layout (see semantics.md § “Foreign rings”), rather than the native LRSRNG1 shape:

The producer side (Proposal B M3a) mirrors these:

Field reads/writes are host-native endianness (the foreign producer and Hale are both little-endian x86-64). v1 is byte_records only; the slots framing kind and a zero-copy writable producer view are post-v1.

v1 scope: single-producer, multi-consumer; in-memory delivery is in scope (POSIX shm_open works intra-machine cross-process). Multi-producer (CAS-based claim), back-pressure / timeout modes, and named-ring registry are post-v1. The Hale-side fallible(ClaimError) signature is reserved for those; v1’s claim() never actually fails.

Lifecycle / cleanup. Both lotus_bus_register_shm_ring (publisher) and lotus_bus_register_subscriber_shm_ring (subscriber) register a single atexit hook on first call. The hook:

  1. Signals every subscriber reader thread to stop via an atomic should_stop flag.
  2. pthread_joins each reader thread, ensuring no in-flight handler is interrupted.
  3. Frees the subscriber state allocated by the registration call.
  4. lotus_shm_ring_closes every ring opened in this process — which shm_unlinks the ones this process created (owns_unlink=1), keeping /dev/shm/ clean across restarts.

The atexit hook runs on a clean process exit (return from main, exit(3)), which includes a SIGINT / SIGTERM that drains to its end (GH #1039). Termination that does not return from main — SIGKILL, a signal the program does not observe, a drain that outlives its grace, _exit — bypasses atexit and leaves the SHM namespace entry behind until reboot or manual shm_unlink.

Constraint: subscriber handlers must not call exit(). The handler runs on the reader thread; calling exit() from inside the handler invokes atexit on the reader thread, which then attempts to pthread_join itself (undefined behavior). Use _exit() if a handler needs to terminate the process immediately.

Codegen surface for K5 lands in K4 (route-selection + slot-locus synthesis) and K6 (subscriber view + epoch guard). K5 ships the substrate; user code can’t reach these symbols directly without going through the slot-locus surface a zero_copy binding produces.

Closure-test infrastructure

Section titled “Closure-test infrastructure”

Perspective infrastructure

Section titled “Perspective infrastructure”

Failure handling

Section titled “Failure handling”

Lifecycle obligations

Section titled “Lifecycle obligations”

The lifecycle is a table of obligations (F.40 phase 3, L1): each action the compiler emits or the runtime performs is owed by a domain to an instance, is placed in order by the events it waits for, and ends in one of its named terminal outcomes. The rows are hale_types::lifecycle (crates/hale-types/src/lifecycle.rs). An obligation is keyed by its source site, the declaration built and the construction template that builds it (P1’s instance key); the runtime numbers the live instance and its incarnation, one per birth(), and the table never does. A restart begins the next incarnation; a restart asked for and not performed begins none.

This section holds the decisions the lifecycle inventory (notes/f40-lifecycle-inventory.md, Decisions 1–19) asked for, one paragraph per line, and the requirements adopted beside them. It sits apart from § “Failure handling” because most of the lines are about order and teardown, not failure; decision L0-1 stays there. Each paragraph states the rule and whether it is shipped. Where the adopted rule is not yet what the code does, the paragraph names the inventory row where they differ, and the fixture under crates/hale-codegen/tests/fixtures/lifecycle/ that pins today’s outcome; lifecycle_fixtures.rs lists it in its KNOWN_OPEN table and fails once the outcome changes, so the entry has to go with the fix. Each fixture also runs under the lifecycle trace (§ “The lifecycle trace”), held to the plan the table’s producer (hale_types::lifecycle::derive) derives for its program, on its line’s rules (one of line 8’s, whose shape the producer does not derive yet, to a hand-written plan). A failure’s delivery is claimed on its owner’s domain, in place or posted there (decision L0-1), and on the raising thread where that domain has already ended (the shutdown rule, join progress below). Occurrences of one declaration whose runs end differently in one execution (one completes, another is canceled on main or refused) are counted from the run’s path, and a main locus built more than once owes its eager spine at each of its teardowns: line 19’s three cross-pool and shutdown fixtures are derived that way. The six started-run retention fixtures use the derived plan, including the edge from each run’s end to its reclaim’s completion. A subscriber a pool hosts owes a Handler row the same way (line 19, inventory row R52): one per delivered cell, as many as the run delivers, its instance retained until each ends, and the reclaim’s completion after the end of every one that began before it (l19_handler_cell_retained.hl); the worker’s hold, which a classic worker may carry to its next cell, ends no earlier, and no later than the carry’s bounds (line 19 below). A posted run may overlap drain and dissolve; an inline run ends before drain. The producer also keeps statement-position subscribers alive until frame exit, where their teardown runs, rather than assigning them an eager teardown at the literal. Any other literal a statement of a lifecycle body builds is torn down where its statement ends, so that body’s row completes after the literal’s reclaim: a run that exits inside the teardown has not reached the body’s end and owes nothing that waits for it. An owner’s Reclaim entry follows its children’s Dissolve completion; its Reclaim completion follows their Reclaim completion. This permits retained storage while preserving physical release from children to owner. A departure the trace shows and the outcome cannot (a missing step, a step on the wrong thread) is in the same file’s TRACE_KNOWN_OPEN table. A line still waiting on a condition says so and records today’s behaviour. The same rules are evidenced across shapes by the lifecycle matrix (crates/hale-codegen/tests/lifecycle_matrix.rs): a generated program for each failure phase, tree position and domain, held to its outcome, the producer’s plan for it and AddressSanitizer, with the cells that fail today, the inventory row each fails at (two, for a cell that shows two known defects), and the departures each shows, in its KNOWN_OPEN table, which is empty today.

The lifecycle trace

Section titled “The lifecycle trace”

A debug aid, not a contract. A build made with HALE_LIFECYCLE_TRACE=1 (BuildOptions::lifecycle_trace; native host targets only) compiles the runtime with LOTUS_LIFECYCLE_TRACE and emits each lifecycle step between an entry call and its completion event, so the program writes one line per obligation event on stderr:

lc <seq> <Kind> <Point> spine=<Spine> dom=<domain> type=<T> inst=<n> inc=<n>

Kind is an ObligationKind, Point is Entered, Completed or Terminal(<terminal>), and Spine is the spine that emitted the step (- where the site cannot name it: a failure delivered in place, a restart’s re-birth). A Restart carries the spine of the step that decided it (the posted run’s loop, PoolRun; a pinned locus’s thread, PinnedMain; the run gate of an instantiation, Instantiation; the resume at settle, Settle), and a Resume the settle’s. The domain is the thread the event ran on: main, pool:<name> for a cooperative pool’s worker, pinned:<n> for a pinned locus’s thread, thread:<n> for any other. type, inst and inc name the subject; a process-level obligation (the pool join, a wait-abort, a pre-drain) prints - for all three.

The runtime mints the subject, and this is the one place a runtime subject comes from: the first event naming a struct gives it the next instance number and incarnation 0, Restart Entered begins its next incarnation, and Reclaim Completed retires the number, so a struct recycled at the same address is a new instance. The events: ParamsSettle (the bracket’s entry, and its settle), Accept, Birth, Run (and Run Terminal(CanceledAfterStart) for a started run whose parked coroutine a pool worker abandons at shutdown, with that Cancellation; a run that never starts ends Terminal(NotStarted(...)): Acknowledged when its child’s Reclaim cancels it, with that Cancellation, inside the reclaim; Shutdown(PoolShutdown) when the post is refused at shutdown, on the posting thread; Shutdown(PoolTeardown), with a Cancellation, when the pools’ teardown frees its cell), Handler (a bus handler on a pool’s worker, once per delivered cell, carried hold or not: entered as the handler starts, holding its subscriber, completed as it returns, the hold still standing, or Terminal(CanceledAfterStart) where a shutdown abandons its parked coroutine; the oracle holds every handler of a subscriber that began before its Reclaim completed to have ended first; a hold carried past that end only makes the reclaim wait longer, and the trace does not name it), FailureDelivery (entered where the failure is raised, completed when the handler returns, in place or at settle), ConstructionDelivery (a held failure, from the hold to its handler’s return at settle), Restart, Resume (a held failure’s resume at settle: its decision to restart, start run() or end, each traced as its own step after it), Drain, Dissolve (the dissolve-epoch closures and dissolve()), Reclaim (past the __arena latch: the queued runs’ cancellation, then the arena’s release), PreDrain, WaitAbort, PoolJoin and PinnedJoin. Readiness, subscription and the run’s admission have no events yet.

The subject table publishes each initialized identity with a release/acquire ready flag: a thread finding a claimed slot waits until its instance number and initial metadata are ready. Later metadata updates and the sequence counter use relaxed atomics. On one thread seq order is program order, and if event a happens before event b then seq(a) < seq(b), so a seq order that contradicts a required edge is a real violation, and one that agrees with it is evidence of an execution, not a proof. The subject publication and the write itself (one write(2) per line) perturb the execution; a traced run does not establish the ordering of an untraced run. hale_types::lifecycle::trace parses the lines back into Events and checks them against what a run owes.

LOTUS_LIFECYCLE_SKIP, read by a trace build’s runtime at start, is a comma list of steps a negative control removes: a kind’s name skips that step and both its events where it is emitted (and, for ConstructionDelivery, holds no failure, so the handler runs in place while the params are open; for Cancellation, a reclaim cancels no queued run, which only a fixture whose cells no worker will dequeue may use); two name a hold, not a kind: RunHold, where a reclaim does not wait for its child’s started runs, and HandlerHold, where a pool worker links no hold on a handler’s subscriber (and so carries none), its Handler events still named; <Kind>.<Point> drops that one line and nothing else. A build without the knob emits nothing of the trace and its IR is the same. lifecycle_fixtures.rs’s CONTROLS use it so that, for every obligation kind a fixture’s plan holds a run to, a run with that step removed or reordered fails the oracle, and with the violation that says why: a removed pool join lets a worker’s teardown begin before its run() has ended, a removed hold delivers a failure before its owner’s settle, a removed handler hold lets a subscriber’s reclaim complete while its handler still runs, an omitted completion leaves a dependent step entered with its prerequisite unreached, and the host’s own order (join, then abort the waits) fails line 7’s edge.

Native observation emission (iris P4, 2026-07-27)

Section titled “Native observation emission (iris P4, 2026-07-27)”

With LOTUS_OBS=1 the runtime publishes an iris-protocol observation segment (/hale-obs-<pid> + registration file per PROTOCOL v0.4 — the contract’s executable form is crates/hale-codegen/runtime/obs_protocol.h in this repo since 2026-09-09 (GH #527 B1); lotus_obs.c is compiled with it prepended and pins its layouts to it, and the Rust test decoder is checked against its text. 0.2, 2026-08-12 downstream handoff: the header gains model_hash at 0x80 — the topology artifact’s shape_hash, stamped by the CLI from the model of the snapshot it typechecks, so a consumer can establish the RUNNING binary was built from the model it joins against; 0 = unstamped harness build. And the per-binding backpressure cells reserved since v0 are now written: queue_depth (cell 3, a last-write-wins gauge of the kernel send-queue occupancy sampled at send time), send_block_ns (cell 4, accumulated transport-send duration), and retries (cell 5, reconnects) — counters-tier, so a consumer falling behind shows as depth climbing and block time accruing BEFORE anything drops) and the Canonical model entity ids (GH #476 Change 8, proto 0.3). Manifest rows name entities and number them in registration order, which is a fact about the run, not about the program. Each row’s aux_b — in the entry layout since v0, written as 0 by every path until now — carries the canonical ApplicationModel entity id for that row, stamped by the CLI at build time: MK_TOPIC rows carry the SubjectId (the manifest fuses publishers by wire subject, so the address, not the topic decl, is the identity), MK_LOCUS_TYPE rows the LocusDeclId, MK_BINDING rows the BindingId. Values are index + 1; 0 keeps meaning “unstamped” (a harness build, or an entity the model does not name — a stdlib subject, say).

Those ids index tables that model_hash does NOT fully cover: shape_hash is structural model identity, and the arrangement’s binding rows are not in the topology artifact at all, so two builds can share a model_hash while numbering entities differently. The header therefore publishes the id table’s own identity at 0x88: entity_id_digest, a digest over the exact (kind, name, id) rows the build stamped. A consumer recomputes it from the model it holds and uses the ids only on a match — a detectable refusal instead of a silent misattribution. 0 = unstamped. A consumer that joined by name still can.

The stamp is all-or-nothing: the mapping table grows with the program (no cap), and if it cannot, the process says so on stderr and publishes NO ids and NO digest rather than a partial table — partial canonicalization is indistinguishable from “unstamped” and would put a consumer back on name matching without telling it.

proto 0.4 (2026-09-04, GH #525 / iris handoff-14 P31). No layout change. Iris’s PROTOCOL §4 and its reference emitters had used aux_b since v0 as binding → owning topic id, scheduler → cpu index, so from 0.3 a nonzero aux_b meant two things depending on which emitter wrote the segment. 0.4 retires the v0 meaning: every emitter writes the canonical entity id or 0, the scheduler cpu index moves to aux_a, and the binding → topic pairing is dropped (the counter table and the binding name carry it). A consumer still gates on entity_id_digest != 0; at minor ≥ 4 it may additionally trust that a nonzero aux_b was never anything else. This was the last open item before the protocol freezes at v0.

Identity is stamped before anything can register. The prelude publishes model_hash, exec_digest, and the entity ids ahead of the bindings prelude and the config loader, because registering a binding creates the observation segment and segment creation snapshots the identity fields — a program with a bindings { } block used to publish model_hash 0 for its whole life. Eager recording/replay init still runs after binding realization, so a backend with no replay class refuses at its own seam, naming itself, instead of being pre-empted by a generic identity refusal.

The runtime’s own choke points emit records: BUS_PUBLISH / BUS_DELIVER at every dispatch flavor (dynamic, static-devirt, cross-thread wire; deliver is enqueue-time at v0), NET_SEND / NET_DELIVER at the transport fanout/reader with per-binding monotonic seqs (what iris seq-matches into cross-process edges), LOCUS_BIRTH / LOCUS_DISSOLVE (both cooperative and pinned lifecycle paths; dissolve rides the emit_locus_arena_destroy chokepoint) and RESTART at reconnect. Cost contract: env unset = one predictable branch per probe (g_obs_state); enabled-but-unobserved (observer_count == 0) = counters only, no ring writes; ring emission is SPSC via the #247 primitive with one ring per emitting thread (TLS assignment; overflow threads count into ring_drops_total). On the observer-count 0→1 rising edge the live-locus table replays as LOCUS_BIRTH records so a late-attaching observer reconstructs the tree. Knobs: LOTUS_OBS_RINGS (default 8), LOTUS_OBS_SLOTS (default 4096), LOTUS_OBS_WIRE (default off — see NET seq semantics: LOTUS_OBS alone never alters the wire; the cross-process edge header rides the wire only when the operator opts the whole fleet in with LOTUS_OBS_WIRE=1). v0 notes: manifest ids are registration-order; publisher locus attribution on BUS_PUBLISH is unattributed (0); pinned births render parent=root. lotus_obs.c is its own TU; the arena TU’s probes are weak-guarded so helper binaries compiling lotus_arena.c alone still link.

Field-hardening (iris handoff 2, 2026-07-27, v0.11.13): NET_SEND now fires on the UDP multicast fanout path (it continues before the stream-transport probe, so multicast publishers emitted only deliver-side records — the seq matcher had nothing to pair); LOCUS_BIRTH is emitted BEFORE a locus’s field-default init and carries real parentage (the parent registers before its children look it up — post-birth placement rendered every tree flat), and pinned children register on the spawning thread so a park before their first probe doesn’t hide them; BUS_PUBLISH stamps the publishing locus from a codegen-set TLS (the C dispatch doesn’t know self); topic shape_hash is fnv(subject, canonical payload structure) — field names + coarse type tags in declared order, never the declaring type’s local name — so two binaries sharing a subject fuse into one manifest row; and each ring re-emits EPOCH every 1024 records so a high-rate ring that wraps its anchor never reconstructs timestamps from a stale base (the ~2^64 ns readings).

NET seq semantics (iris handoff 3, v0.11.14): NET_SEND and NET_DELIVER carry w1 = origin:16 | seq:48 where origin is the SENDER’s per-process identity and seq is the SENDER’s per-binding counter — the receiver echoes both verbatim from the wire, so a send and its delivers pair on (origin, seq) across segments (the cross-process edge). Before this, the receiver stamped its LOCAL delivery counter, which sums across senders on a multicast subject — the send seq never equalled the deliver seq and iris rendered zero edges. (Stream transports are unicast — one sender per connection — so origin 0 + the framed wire seq already pairs correctly.)

Edge-emission correctness (iris handoff 4, v0.11.15):

Flavor completeness + quiet-process replay (iris handoff 5, v0.11.18):

BUS record w1 layout (iris handoff 7, v0.11.21): BUS_PUBLISH / BUS_DELIVER pack w1 = locus:20 (bits 44..63) | seq:44 (low) — PROTOCOL §8, executable reference iris emitter/protocol.h (obs_bus_w1). The emitter had these fields transposed (locus low, seq shifted) since handoff-3; attribution was computed correctly and packed unreadably, and the contract tests stayed green because they decoded with the emitter’s own layout. The tests now vendor protocol.h’s decode, so emitter and consumers cannot disagree silently.

Adapter ingest completeness (iris handoff 8, v0.11.22): the Hale-owned-wire ingest (std::bus::__local_dispatch → lotus_bus_dispatch_wire_inbound) now carries the full probe trio the C reader threads have. The wrapper peels the self-describing obs wire header when the producer emitted one (magic-guarded — a headerless or non-Hale producer is byte-for-byte unaffected, and headered bytes reaching a Hale adapter previously failed deserialization outright), emits NET_DELIVER echoing the wire (origin, seq) (or (0, local per-subject seq) when headerless — countable, not pairable), and the plain dispatch_wire fanout gains the per-target BUS_DELIVER its keyed sibling received in v0.11.18. Adapter bindings register lazily in the manifest (aux = 2). Dynamically-ingested planes now form producer→consumer edges and attribute subscriber loci exactly like statically configured listens.

Attribution ordering + the adapter inbound path (iris handoff 6, v0.11.20):

Lossless recording mode (GH #296 Phase 1)

Section titled “Lossless recording mode (GH #296 Phase 1)”

LOTUS_OBS_RECORD=<path> opts a run into recording: the observation plane stops being a sampler and becomes a flight recorder. Implies LOTUS_OBS=1; the recording counts as an attached observer, so ring emission is on from the first probe with nothing mmapping the segment. Opt-in on the same terms as everything else here — unset, the lowering and the wire are byte-for-byte the unobserved build (the flag resolves in the same constructor as lotus_obs_live and gates behind it everywhere).

Three deltas from observer mode, all in the service of one rule — a recording never drops (a dropped record is a replay that diverges silently):

Durable recording / crash-prefix recovery (Phase 5). Named precisely: this is a durable flight recorder, NOT a write-ahead log — the application is never gated on the recording reaching stable storage, so a crash can lose records the application already executed past. (A load-bearing WAL grade — persist, fence, then permit delivery — is a possible later mode; nothing here claims it.) What IS guaranteed: the drain appends whole frames in stream order and flushes every sweep, so the on-disk file is an exact prefix of the stream at all times, and the header identity (model_hash, exec_digest, policy flags) is stamped eagerly by the drain as soon as the ctor-sequenced setters have run, not only at finalize. A run that dies without teardown therefore leaves an artifact that is attributable and exact up to one torn frame at the tail. Default recording rides the page cache (survives a process crash, not power loss). LOTUS_OBS_RECORD_DURABLE=1 is the power-loss grade: every flushed sweep synchronizes the file (fdatasync on Linux, F_FULLFSYNC on macOS), the parent directory is synced at file creation (a fresh NAME is not durable until its directory entry is), the finalize trailer itself is synchronized before close (a clean exit followed by power loss must not demote a finalized recording to a truncated one), and the grade is recorded in the header’s policy flags (bit 1).

Consume records (private recorder namespace). BUS_DELIVER is enqueue-time by design and lands on the publisher’s ring — correct for fanout accounting, structurally unable to say in what order a consumer actually ran its handlers. Under recording, every dequeue-driven handler invoke (main-queue drain, coop-pool drain, pinned mailbox drain, async coro start) stamps a consume record on the consuming thread right before the handler runs, carrying the delivery identity: the target locus in the record’s id field and the full 64-bit msg_id in w1. Its ring position is the per-consumer delivery order — the thing a replay serves back. Pairing rule: per queue, the k-th consume is the k-th queued delivery. The synchronous direct-dispatch flavors deliberately emit no consume record: their handler runs at the BUS_DELIVER position, which is the consumption point.

Recorder events (CONSUMER/CONSUME/ENQ/JOURNAL) live on process-private per-thread rings in their own event namespace, never in the public observation segment: iris protocol ekinds 8/9/11/12 mean SUPERV_TRANS/PLACEMENT/BINDING_UP/BINDING_DOWN, and an observer attaching to a recording process must never decode recorder bookkeeping as lifecycle transitions. In the recording file, private-ring entries carry the high bit in their ring field, so the two namespaces can never be confused.

Delivery identity (Phase 2). Every queued delivery carries a deterministic pub_id = consumer_id:16 | per-publisher-thread seq:48 (full width — the review-round guards fail loudly rather than wrap), re-derivable by a re-executed run without global coordination. Consumer ids are stable across runs, unlike pthread ids: main = 1, cooperative pool workers = 16 + registration index, pinned locus threads = 64 + obs instance id; threads with no stable identity (ingress readers) get run-unique anonymous ids, never a shared fallback. Payload bytes are captured once per publish — every dispatch flavor, queued or synchronous direct — under a stable subject hash (manifest topic ids are registration-order and racing publishers register in either order). Flags: bit 0 = external wire ingress; bit 1 = raw in-process struct bytes — an ABI snapshot (String/Bytes fields are pointers, padding is uninitialized) that consumers must compare by size only; canonical per-topic recording codecs are the staged fix. Wire captures are canonical bytes, unflagged. The direct-dispatch flavors — the devirtualized same-thread call that replaces the enqueue, in both its baked-inline and helper forms — record through the same writer, at the same publish site, in the same framing. They used to capture nothing, on the reasoning that a closed-world same-thread call carries no external input and re-execution re-derives its payloads: it does re-derive them, which is precisely why --diff can compare them, and a fully direct-dispatched workload recorded none to compare. The capture sits behind the same recording gate as the publish probe, so an unrecorded run writes nothing and pays nothing.

Input journal (Phase 3). Under recording, every user-facing nondeterministic read is journaled per consumer as ONE unified stream carrying the read’s kind and its exact encoded arguments (length-framed; replay memcmps them — hashes proved collision-prone), so a changed env name, arg index, rand bound, or read length is the first named divergence, never a silently substituted value. Env values are withheld by default: names, existence, and lengths are recorded; the value itself requires LOTUS_OBS_RECORD_ENV=full, the artifact header says which policy applied, and a withheld read replays as a named withheld divergence. The interposed set: std::time::now, std::time::monotonic[_ns] (now a named primitive, lotus_time_monotonic_ns — it used to be inline clock_gettime IR that nothing could interpose), std::rand::next_int (per call — the global RNG’s mutex makes seed-replay unsound across threads), std::os::getrandom, and the std::env surface. Internal runtime clocks and config getenvs are deliberately not journaled.

File format v0.3 (PRE-STABLE). 96-byte header (magic HALEREC0, version, pid, ring geometry, epoch anchors, model_hash, a 32-byte framed-SHA-256 exec_digest, and policy flags), tagged entries — tag 0: 24-byte ring record; tags 1/2/3: payload / journal / meta blobs (32-byte header + bytes padded to 8; meta carries the run-stable topic and public-ring identity maps) — and a 16-byte trailer (HALEEND0 + entry count). A recording is clean only when the ENTIRE artifact validates: exact parse to the trailer position and a matching entry count, enforced by the CLI reader and independently by the runtime loader (one open + fstat + private mmap, checked arithmetic, per-kind value shapes) — trailer magic at EOF alone proves nothing. Identity fields are stamped eagerly by the drain and re-stamped at finalize (a prelude binding registration can probe — creating the header — before the stamps run; the two stamps agree). A trailer-less artifact is a crash-truncated recording: refused by default, but admissible as a PREFIX with hale replay --allow-truncated (LOTUS_REPLAY_ALLOW_TRUNCATED=1) — the loaders stop at the first incomplete frame, report the parsed extent and dropped tail bytes, and replay that prefix; execution past the tape’s end falls back live and is counted, per the degrade-never-refuse rule. Under --diff a truncated baseline compares as a prefix — and the runtime verdict agrees with the comparator: recorded-history-exhausted events (journal reads past the tape, deliveries past the consume stream, consumers the recording never saw) count into a separate post_prefix_live_fallback status key, never into the divergence totals; mismatches BEFORE exhaustion stay divergences either way. A trailer-FINALIZED artifact keeps the exact full-parse + count checks: with a finalize present, any truncation is corruption.

Replay (hale replay <recording> <program.hl>). Admission, strongest check first: the recording’s exec_digest — a framed SHA-256 over the toolchain source hash (compiler + runtime + stdlib implementation, via the stale-CLI build hash), the CLI version, build options, and every source file’s path, length, and contents — must match the recompiled program exactly. A file’s path is its path in the program’s source map, in the map’s order (F.40 phase 4, I3; verification.md § Source maps and model semantics): relative to one root, the same whichever target loaded the program, wherever the tree is checked out and however the target is typed, and distinct for two files of one name; the build options are the ones the compiling command was GIVEN (hale run and hale replay take hale build’s option flags — 2026-09-20, GH #904; run compiled with the defaults and fingerprinted the defaults), so a recording made under hale run --dev is admitted by hale replay --dev and by no default replay. hale build, hale run and hale replay compute the options one way (F.40 phase 4, I2): the command’s flags, --env‘s role table (replay --env resolves the environment as run --env does, so a recording made under --env prod is admitted by hale replay --env prod and refused under another environment’s roles or none), and the [ffi] link libraries and C sources each imported package’s hale.toml declares, which every one of the three folds in and builds with, so a program that needs a package’s [ffi] csrc runs and replays without --csrc / --link flags, and a flag naming what a package’s [ffi] names is a no-op that leaves the identity the flagless invocation’s. Debug information is not part of it: the DWARF line tables hale build adds by default change no behaviour, so a recording made by a binary from hale build of a file is admitted by hale replay of that file, and a built binary and hale run of one program carry one identity. A recording of a DIRECTORY build is admitted by the replay of its entry file: the two name each file by its source-map path, so they frame one program alike. The build options include the environment’s build knobs that change the binary (see Build-time and toolchain environment: LOTUS_ASAN, LOTUS_TSAN, LOTUS_UBSAN, LOTUS_LTO, LOTUS_DISABLE_PREFETCH, LOTUS_NO_BUS_DEVIRT, LOTUS_NO_OWNERSHIP_BUBBLE, HALE_NO_TS_SHIM, HALE_TS_SHIM_A, HALE_ZIG, HALE_TARGET_GLIBC, HALE_TARGET_SYSROOT, LOTUS_OPENSSL_PREFIX), so a recording made under one is admitted by no replay built without it; what only narrates or times a build, the warnings, the linker and the cache directory are not part of it; shape_hash is structural compatibility only, the secondary check. An unstamped recording is refused without --allow-unverified-model. Stated residue: the digest cannot see the LLVM/libc toolchain outside the hale binary; a post-link binary digest is the staged stronger form. Safe by default: replay re-executes real side effects, so the typed effect rows gate admission and fail CLOSED on syscall, ffi, unclassified (“may do anything”) — refused without an explicit --allow-live-effects (coarse by class granularity — over-refusal is the safe direction; per-primitive replay classes are the staged refinement). bindings blocks are no longer in that list: under replay the wire is hermetic (below), so a bound topic cannot reach the live world. LOTUS_REPLAY=<path> then serves journaled reads back per consumer in recorded order; a read past (or mismatching) the recorded history falls back live and is counted — replay degrades, never refuses (RFC Q6) — with a divergence summary at exit and a machine-readable verdict (LOTUS_REPLAY_STATUS) the CLI consumes: journal misses, order holds, unconsumed journal entries, and unconsumed deliveries all count, and any of them fails --diff. Each consumer’s queued deliveries are re-consumed in the RECORDED order (Phase 4): dequeued cells that arrive ahead of their recorded turn are held per-consumer and released in order, with a bounded hold (1s) after which the oldest held cell is released and the miss counted, so a genuinely divergent replay reports rather than deadlocks. A job posted to a pool is a queued cell of its own and so a recorded start: a consume of delivery identity 0 whose target is the locus it starts, matched by the gate against a recorded 0 slot. A pool-placed root posts three, each with its own slot in posting order — its initialization, its own birth step (decision line 3) and its run() — and a run() posted by any other child is one more; a replay consumes them in that order. A run its child’s reclaim canceled in the queue (decision line 19) is dropped before the gate compares it, as the recording dropped it, with no consume; a live run the gate holds keeps its retention on the child and is admitted only when it is dispatched. The hold belongs to its consumer thread for the thread’s life: a pool worker or a pinned thread frees it when it exits at the pools’ or the owner’s join, and a cell found still held then ends as one the pools’ teardown frees undequeued, never dropped with the hold.

Async pools replay (Phase 6). The nondeterminism of a where async_io pool is its drain’s SCHEDULING: which cell starts when, which parked coro resumes when (epoll readiness order), and when a timed park expires relative to both. Under recording, every such decision is stamped on the pool worker’s private ring as a step (ASYNC_START/ASYNC_RESUME/ASYNC_EXPIRE; coros are named by birth ordinal — the index of the START that created them, stable across runs because start order is enforced). Under replay the drain is DRIVEN by that stream instead of the clock: START steps reuse the phase-4 cell gate (start order is consume order); RESUME steps wait for the named coro’s readiness, parking early-ready coros aside until their turn; EXPIRE steps resume the named coro with the timed-out sentinel immediately — the recording already proves the deadline fired at this point in the sequence, so replayed sleeps fast-forward rather than re-waiting. A step unsatisfiable within the hold bound counts an async-schedule divergence and is skipped (degrade, never deadlock) — and a skipped START retires both its birth ordinal and its consume slot (review round 2: ordinals belong to the recorded START slots, not to whichever cell happened to start next — without retirement one missing delivery shifts every later coroutine into an earlier recorded identity, and the pinned consume expectation makes every later START mismatch; with it, later steps that name a retired slot skip immediately and the divergence stays local).

Coverage, named (review round). A dry tape hands the pool back to the live drain — never silently. The artifact carries a capability bit (header flag 4: this runtime records async schedules); the states are distinct and reviewable: a pre-phase-6 artifact gets a one-shot coverage note (“recording predates async-schedule support”); a truncated tape’s remaining schedule is the stated coverage boundary (excluded from the divergence totals); a FINALIZED capable artifact whose tape runs dry while the re-execution keeps doing async work counts each such action as async_post_tape, and schedule steps left unconsumed at exit count as unconsumed_async_steps — both machine-readable status keys and divergence rows. --diff compares the per-consumer async step streams (kind, value) bidirectionally, exactly like the consume and journal streams — “byte-identical output” alone is weaker than “the schedule matched”. Boundary, stated: the SCHEDULE replays; the DATA of unjournaled I/O does not — a coro resumed at its recorded turn re-executes its recv against the live world (syscall-class, gated), and bindings ingress arrives via the Phase-5 injector.

Hermetic wire + ingress injection (Phase 5). Hermeticity is a binding-kind capability, not a blanket assumption: the native unix:// and udp:// transports (and the transport-locus form) are suppressed and injectable; a backend with no replay class fails closed — shm_ring in particular is refused by the CLI (backend named) and independently at the runtime’s shm-open seam, so a replayed or fed process never creates, opens, or mutates live shared memory. Adapter loci are user logic and re-execute as such (governed by their inferred effects).

For the covered backends, under replay a binding is suppressed at realization: the entry exists (a husk, transport NULL — the same shape the reclaim path already leaves), but no socket is created, nothing binds, nothing connects, and outbound fanout to bound subjects sends nothing (the publish is still recorded and compared under --diff).

In the listeners’ stead, injection runs as an explicit boot phase: codegen emits one lotus_replay_start_ingress() call at the main locus’s boot/run boundary — params children born, bindings realized or suppressed, every boot subscription registered, run() not yet entered — where the runtime snapshots the subscription registry ON the main thread (injector workers never read the live registry) and spawns one worker per recorded ingress source (the pub_id’s consumer bits), each carrying its source’s recorded consumer identity so a --diff verify recording aligns per-consumer streams with the original instead of collapsing every listener onto one injector identity. Fresh anonymous claims are floored above the recorded range. The snapshot IS the coverage boundary: a subscriber the program only creates during run() (spawned or accepted) is not visible to injection — such tape entries classify as late_subscription_uncovered (the injector join rescans the live registry at teardown to distinguish them from genuinely absent subscribers), a stated boundary rather than a silent unmatched. Injection is keyed by full identity: each tape record carries its complete subject string and the subject’s canonical payload shape (FNV-32 alone is collision-prone and is kept only for reporting); a record whose shape does not match the live program’s shape for that subject is refused as incompatible, never fed as a plausible wrong value. Only messages the original application accepted enter the tape — the wire capture runs after the deserializer says yes, so identity allocation agrees between record and replay and rejected traffic is never re-fed.

Pacing. Strict replay injects one message, then waits (bounded by the phase-4 hold) until that message’s recorded consumes have all been re-consumed — unpaced injection can shed at bounded queues (bounded(N, drop_old/drop_new), on_full policies) before the dequeue-side order gate ever sees the cell, which no amount of holding can undo. Feed mode is deliberately unpaced.

Accounting. Every tape entry ends in exactly one class: injected, rejected-by-deserializer, unmatched-subject, incompatible-shape, or unprocessed-at-shutdown (the injector is joined at teardown and its remainder classified); injector start failure is its own class. Under strict replay every non-injected class is a divergence, in the machine-readable status (ingress_*, injector_start_failure keys) and the exit summary.

Feed mode (hale replay --feed, LOTUS_REPLAY_FEED=<path>) — same recorded ingress, changed code, live nondeterministic environment. Stated that precisely because “same inputs” would overclaim: feed reproduces recorded binding ingress only — time, randomness, env, and argv stay live. The recording is consumed as an input tape: recorded ingress injected, wire hermetic, and nothing else of replay applies — no journal serving, no order enforcement, no model admission (a model-hash mismatch is reported informationally; feeding a tape to changed code is the point), no --diff/--at (there is no recorded schedule to compare against). The effects gate still applies: --feed bypasses identity admission, never effect safety — a program whose frontier reaches syscall/ffi/unclassified still requires --allow-live-effects alongside --feed. Mutually exclusive with LOTUS_REPLAY. Injected deliveries carry the new run’s own identity. The exit report classifies every tape entry, and an unfed remainder (unmatched, incompatible, unprocessed, start failure) fails the run by default — --allow-unmatched-feed is the explicit acceptance of a partial feed.

--diff (strict replay only — feed rejects it, above). --diff records the replay (under the original’s env policy) and compares bidirectionally: per-consumer queued consume streams (target locus + msg_id), per-consumer PUBLIC bus streams aligned by subject and stable consumer id via the meta maps — which is what makes synchronous direct dispatch visible — payloads both ways (topic, flags, bytes; raw metadata by declared size), and per-consumer journal streams (kind, exact args, withheld state, value; grouped per consumer because the drain interleaves concurrent threads’ entries in incidental order). Any runtime divergence fails --diff through the verdict. --at <n> stops (SIGSTOP) at the nth consume process-wide (meaningful for one consumer); --at <consumer-id>:<ordinal> is the stable multi-consumer form. Replay implies observation (identity rides the obs machinery).

What a match reports — coverage, per category (GH #728). A match is only as strong as the categories the recording carries observations for, so the success report names all five and states, for each, either the count it compared or that the category was not exercised and why: public bus events (with their consumer count), payloads, queued consumes, async schedule steps, journal reads. A category with no recorded observations is never printed as a compared zero — “0 consumes across 0 consumers” read as a verified queued delivery schedule when the workload direct-dispatched every delivery and no queued schedule existed to verify (and the same template asserted payload identity for a recording with no payload blobs). Direct dispatch stays valid and is named as such: its deliveries are compared, as public bus events, and its payloads are compared with them — a direct-dispatched publish records its payload blob like any other (above), so payloads is a compared category for such a workload, not an unexercised one. The payload line says what was compared, per the blobs the recording carries (GH #842): an in-process flat payload’s blob is metadata only (flag bit 1, above), so for those the comparator checks declared size and publish identity and never the contents, and the report reads payloads: N (sizes and identities; contents not canonicalised); wire captures are canonical bytes and read canonical bytes identical; a recording carrying both counts each. A flipped field inside an in-process payload is therefore NOT a divergence --diff can see today — the per-topic canonical codec that would make it one is a separate feature item (GH #947), not part of this report. Coverage is derived from the recording the comparator walks and gated on the same async-capability bit, so the report cannot claim a category diff skipped. --json (strict replay with --diff) prints the same verdict machine-readably — result, ring_records, recorded_prefix_only, and per category compared, count, consumers, not_exercised_because, and for payloads the same split as contents_canonicalised and metadata_only. Success semantics are unchanged: a divergence in any compared category still fails, and a diverged verdict carries the reason with no per-category counts (the comparison stopped at the first difference).

Two honest limits, stated rather than implied: the recorded interleaving is reproduced per consumer, not globally — cross- consumer wall-clock alignment is not a replay property; and the injected tape covers bindings ingress — raw user-level socket reads are syscall-class live effects (gated), and adapter loci re-execute their own protocol logic. Fleet-scale replay (multiple binaries against one composed tape, with #262’s plan admission for replay-under-a-different-plan) is the next milestone.

Time

Section titled “Time”

I/O — minimal

Section titled “I/O — minimal”

Text + string primitives (v1.x adds)

Section titled “Text + string primitives (v1.x adds)”

Process control

Section titled “Process control”

stdout buffering

Section titled “stdout buffering”

stdout is line-buffered for the lifetime of the program, regardless of whether it’s attached to a TTY or a pipe. The main prelude calls setvbuf(stdout, NULL, _IOLBF, 0) once before any user code runs.

The default libc behavior (fully-buffered when stdout isn’t a TTY) silently dropped output for any program that printed then blocked on a syscall — println("READY"); accept_loop(); made “READY\n” invisible to a piped consumer until the buffer filled or the program exited. Test oracles, supervisors waiting for a READY handshake, and log tailers all hung. Line-buffering matches Python python -u discipline and Go’s default; \n- terminated println calls flush immediately under any stdout target.

stderr is line-buffered by POSIX already; the runtime doesn’t touch it.

What’s NOT in the runtime (lives in stdlib instead)

Section titled “What’s NOT in the runtime (lives in stdlib instead)”

These are bundled with the toolchain (no separate install) but require explicit import std::....

Form-vec runtime (v1.x-FORM-1)

Section titled “Form-vec runtime (v1.x-FORM-1)”

The @form(vec) form lowers to a contiguous growable buffer implemented in C. See spec/forms.md for the form contract and synthesized method set; this section documents the runtime shape.

C struct layout

Section titled “C struct layout”

Each @form(vec) locus’s heap slot lowers to an inline struct:

typedef struct {
size_t cap; // allocated capacity (elements)
size_t len; // number of valid elements
char *buf; // contiguous element array
} lotus_vec_<T>_t;

The <T> suffix is conceptual — codegen monomorphizes per cell type T, but the runtime primitives operate on the common prefix layout via void * casts. All lotus_vec_*_t typedefs share the {cap, len, buf} prefix.

Primitive functions

Section titled “Primitive functions”

Defined in crates/hale-codegen/runtime/lotus_arena.c (v1.x-FORM-1 PR4):

Function Behavior
void lotus_vec_init(void *v) Zero-init: cap=0, len=0, buf=NULL
void lotus_vec_push(void *v, size_t es, const void *x) Append; doubles cap on overflow
int lotus_vec_get(void *v, size_t es, int64_t i, void *out) Bounds-checked read; returns 1=OK, 0=out-of-bounds
int lotus_vec_set(void *v, size_t es, int64_t i, const void *x) Bounds-checked in-place write; returns 1=OK, 0=out-of-bounds (does not extend the vec)
int lotus_vec_pop(void *v, size_t es, void *out) Returns 1=OK, 0=empty
int64_t lotus_vec_len(void *v) Element count
int lotus_vec_is_empty(void *v) 1=empty, 0=non-empty
void lotus_vec_destroy(void *v) free(buf); called at locus dissolve
void lotus_vec_sort_int(void *v) In-place ascending sort of an int64_t-cell vec via qsort
void lotus_vec_sort_float(void *v) In-place ascending sort of a double-cell vec; NaN treated as equal-to-anything
void lotus_vec_sort_string(void *v) In-place ascending sort of a char *-cell vec under strcmp ordering
void lotus_vec_sort_by(void *v, size_t es, int (*cmp)(const void *, const void *, void *), void *cookie) qsort_r wrapper; cmp is a codegen-synthesized per-(cell_type, direction) trampoline

es (elem_size) is the cell type’s size in bytes — codegen passes sizeof(T) at each call site.

Growth policy

Section titled “Growth policy”

Failure shapes

Section titled “Failure shapes”

Form-hashmap runtime (v1.x-FORM-4)

Section titled “Form-hashmap runtime (v1.x-FORM-4)”

The @form(hashmap) form lowers to an intrusive open-addressing hash table implemented in C. See spec/forms.md for the form contract and synthesized method set; this section documents the runtime shape.

C struct layout

Section titled “C struct layout”

Each @form(hashmap) locus’s pool slot lowers to an inline struct:

typedef struct {
size_t cap; // power-of-two slot count
size_t len; // live entry count
size_t key_size; // sizeof(K), set at init
size_t value_size; // sizeof(S), set at init
int key_type_tag; // 0 = Int, 1 = String
char *slots; // cap * (1 + key_size + value_size) bytes
} lotus_hashmap_t;

Each slot is 1 + key_size + value_size bytes:

[occupied: u8] [key: key_size bytes] [value: value_size bytes]

occupied = 0 means empty. Backward-shift deletion (no tombstones) — probes terminate at the first empty slot.

The C ABI is type-erased: codegen passes key_size / value_size at init time, and per-call sites pass raw void * key/value pointers. Codegen GEPs the indexed-by field on the caller’s side to derive the key pointer before each set.

Primitive functions

Section titled “Primitive functions”

Defined in crates/hale-codegen/runtime/lotus_arena.c (v1.x-FORM-4 PR4):

Function Behavior
void lotus_hashmap_init(void *m, size_t key_size, size_t value_size, int key_type_tag) Allocate cap=8 slots, zero them; freeze key/value sizes and key-type tag
void lotus_hashmap_set(void *m, const void *key, const void *value) Insert or replace; grow at load factor 0.7
int lotus_hashmap_get(void *m, const void *key, void *out_value) Bounds-checked read; returns 1=OK, 0=missing_key
int lotus_hashmap_has(void *m, const void *key) 1=present, 0=missing
int lotus_hashmap_remove(void *m, const void *key) 1=removed, 0=missing
int64_t lotus_hashmap_len(void *m) Live entry count
int lotus_hashmap_is_empty(void *m) 1=empty, 0=non-empty
void lotus_hashmap_destroy(void *m) free(slots); called at locus dissolve

Key types and hashing

Section titled “Key types and hashing”
key_type_tag Type Hash function Equality
0 (LOTUS_HASHMAP_KEY_INT) int64_t Knuth multiplicative (k * 0x9E3779B97F4A7C15) == on i64
1 (LOTUS_HASHMAP_KEY_STRING) const char * (NUL-terminated) FNV-1a over the bytes strcmp == 0, with pointer-identity fast path

Other key types (Bytes, custom structs, enum tags) are not supported at v1; codegen rejects @form(hashmap) with a focused diagnostic when the indexed-by field’s resolved type doesn’t map to one of these two tags.

Growth policy

Section titled “Growth policy”

Deletion policy

Section titled “Deletion policy”

Backward-shift deletion (no tombstones). After clearing the target slot, the runtime walks forward through the cluster and shifts any entry whose natural position is “before” the freed slot in the probe sequence. The cluster boundary is the first empty slot encountered. This keeps probe chains tight and lets find_slot terminate correctly without a separate tombstone marker.

Failure shapes

Section titled “Failure shapes”

Form-ring-buffer runtime (v1.x-FORM-5)

Section titled “Form-ring-buffer runtime (v1.x-FORM-5)”

@form(ring_buffer, cap = N) lowers a pool capacity slot to an inline lotus_ring_buffer_t and synthesizes a fixed-capacity FIFO surface (push / pop / len / is_full). The cap is baked in at lotus_ring_buffer_init from the form annotation arg; the backing buffer is malloc’d once at locus birth and never grows.

C struct layout

Section titled “C struct layout”
typedef struct {
size_t cap; // fixed at init; never changes
size_t head; // index of oldest element (next pop)
size_t len; // current element count, 0..=cap
size_t elem_size; // bytes per element
char *buf; // cap * elem_size bytes
} lotus_ring_buffer_t;

Codegen emits the matching LLVM inline struct on the locus’s pool slot; the slot’s struct field IS the ring buffer (no indirection). Element-size is sizeof(T) from the cell type’s LLVM size_of.

Primitive functions

Section titled “Primitive functions”

Defined in crates/hale-codegen/runtime/lotus_arena.c (v1.x-FORM-5):

Function Behavior
void lotus_ring_buffer_init(void *rb, size_t cap, size_t elem_size) malloc(cap * elem_size); head=len=0
int lotus_ring_buffer_push(void *rb, const void *src) 1=pushed, 0=full; wraps modulo cap
int lotus_ring_buffer_pop(void *rb, void *out) 1=popped, 0=empty; advances head
int64_t lotus_ring_buffer_len(void *rb) Current element count
int lotus_ring_buffer_is_full(void *rb) 1=full (len==cap), 0=not
void lotus_ring_buffer_destroy(void *rb) free(buf) at locus dissolve

Failure shapes

Section titled “Failure shapes”

Native codegen defaults

Section titled “Native codegen defaults”

What the compiler emits for a native hale build:

Build-time and toolchain environment

Section titled “Build-time and toolchain environment”

Codegen reads no environment variable. Every knob that changes what a build emits is a field of hale_codegen::BuildOptions, and the one function that turns the process environment into those fields is build_options_from_env (crates/hale-cli/src/build_env.rs); every command that compiles (hale build, run, test, replay) starts from it, so a variable means the same thing to all four. A caller of the library (a test, another tool) sets the field and touches no environment, and must choose a cache_dir: BuildOptions has no Default, only BuildOptions::new(cache_dir), because the runtime-object cache is the one setting with no right answer to guess; crates/hale-codegen/tests/codegen_reads_no_environment.rs fails if a read appears in the crate.

A boolean variable is on for 1, true or TRUE and off for anything else. A variable marked set is on by being set to any value at all. The rows with no field are read by the rest of the toolchain, not by a build.

Variable BuildOptions field Effect Default
HALE_DEV (set) dev_profile The --dev profile: LLVM O1 module pipeline and Less machine codegen, trading runtime speed for build latency. off
LOTUS_LTO lto 1, true or full: full LTO; thin: ThinLTO; anything else, off. Native, non-sanitizer builds only (see Native codegen defaults). off
LOTUS_ASAN asan Build with AddressSanitizer and skip the O3 module pipeline, so a leak or use-after-free report carries accurate frames. off
LOTUS_TSAN tsan Build with ThreadSanitizer for the runtime compile and the link; the -Wl,--wrap shims are left out. off
LOTUS_UBSAN ubsan Build with address and undefined-behavior sanitizers, aborting on the first UB (-fno-sanitize-recover=all). off
LOTUS_DUMP_IR (set) dump_ir_beside_output Write the pre-optimization LLVM IR to <output>.ll. A library caller may instead name the path in dump_ir. off
LOTUS_NO_BUS_DEVIRT no_bus_devirt Force the all-dynamic bus lowering: every subject’s dispatch plan is empty. The differential harness’s control arm. off
LOTUS_NO_OWNERSHIP_BUBBLE no_ownership_bubble Force the pre-bubble ownership lowering: no bubble plans, no forwarding sets, no threading fields. off
LOTUS_DISABLE_PREFETCH disable_prefetch Compile the runtime without its prefetch hints. off
LOTUS_DI_TRACE (set) di_trace Narrate debug-location decisions on stderr. off
HALE_DISPATCH_TRACE dispatch_trace Print the flavor the bus dispatch plan chose for each subject, with its gate’s payload_flat column, and the codec’s flatness at each publish to a literal subject, on stderr. off
HALE_LIFECYCLE_TRACE lifecycle_trace The lifecycle trace: one line per obligation event on stderr (§ “The lifecycle trace”). A debug build; native host targets only. off
HALE_TIME (set) time_phases Print per-phase wall times of the build on stderr. off
HALE_CC_WARNINGS cc_warnings Let the runtime’s C warnings through instead of -w. For work on the runtime itself. off
HALE_NO_LLD no_lld Link with the default linker even when ld.lld is on PATH (Linux only; lld is otherwise used). off
HALE_NO_TS_SHIM no_ts_shim Behave as if libhale_ts_shim.a was never built: a std::ts program gets the located refusal, other programs link without it. off
HALE_TS_SHIM_A ts_shim The tree-sitter shim staticlib to link, ahead of <hale binary dir> and the workspace target/ dirs. unset
HALE_ZIG zig The zig binary a cross build (--target) compiles and links with. zig on PATH
HALE_TARGET_GLIBC target_glibc The glibc version a cross build’s Linux gnu binary asks for. 2.31
HALE_TARGET_SYSROOT target_sysroot The sysroot a cross build finds OpenSSL, zlib and the shim in. <cache>/hale/sysroot/<triple>
XDG_CACHE_HOME, HOME cache_dir Where compiled runtime objects are cached, content-addressed: $XDG_CACHE_HOME/hale/runtime, else ~/.cache/hale/runtime. An empty value is skipped. else <tmp>/hale-runtime-cache-<pid>, a directory of that process’s own
LOTUS_OPENSSL_PREFIX, OPENSSL_ROOT_DIR openssl_prefix macOS: a Homebrew OpenSSL prefix (the first whose include/openssl/ssl.h exists) for the link. the standard brew locations
LOTUS_NO_DEBUGINFO none: the CLI supplies no debug sources Opt out of DWARF line tables for the Hale code (the runtime C always carries -g). off
HALE_API_TOKEN none The bearer hale watch, admin and mcp --app present to an HTTP or hub exposure when --token is not given; empty is unset. no bearer
HALE_API_BEARER none The bearer hale api describe and hale api call (and their short forms hale describe, hale call) present to an HTTP exposure when --bearer is not given; over a socket the caller is the peer’s kernel credentials and the variable is ignored; empty is unset. no bearer
HALE_BIN none The hale binary a child process runs as its toolchain: hale dna sets it, to the binary it is running as, for the hosts and fixtures it starts. the running binary
HALE_IMPORT_DEBUG (set) none Trace import resolution on stderr, per call. off
HALE_MCP_ROOT none hale mcp: every path a tool call names must resolve under this directory. unset: no restriction
HALE_MODEL_TRACE none Print the model builder’s and the fleet lowering’s demand-proof line on stderr (1 turns it on). off
HALE_REPLAY_TEST_HOLD none A test hook: hale replay waits, between admitting a recording and starting it, until this path exists, so a test can replace the recording at a chosen moment. unset
HALE_SKIP_STALE_CHECK none Any value but empty or 0 skips the check that the binary is not older than the workspace it was built from: every source of the identity-covered crates, the lock file and the ts-shim manifest (statted on each invocation, read only when one is newer than the binary), and the dna/ tree it embeds (statted the same way, digested only when a file or its directory is newer than the binary or the file count is not the embedded set’s). off
HALE_STALE_DNA_ROOT none The tree the stale-DNA check compares the embedded DNA against; the regression test’s way to hand it a tree it may edit. the workspace the binary was built from
HALE_TEST_JOBS none How many workers hale test runs when -j is not given. one per available core
HALE_TEST_KEEP_VAULT none 1 keeps each test file’s vault directory (mode 0700 under <tmp>/hale-test-vaults-<uid>) after its run and prints where, instead of removing it. remove
HALE_WARM_SKIP_IRIS none scripts/warm-dna-cache.sh skips building the observer (1), for jobs that never run hale iris. off

Diagnostic + tuning env vars

Section titled “Diagnostic + tuning env vars”

Every variable the runtime (lotus_arena.c, lotus_obs.c) and the standard library read at run time, and the ones the CLI sets for the process it starts. Unset (the default) keeps the runtime quiet and its behavior as described in this document.

Env var Default Effect
LOTUS_ARENA_LOG_BIG_CHUNKS=<N> off Logs every arena chunk + libc allocator (malloc / realloc / calloc / mmap) >= <N> bytes to stderr with size, monotonic seqno, and 8-frame backtrace. Use 1 (= 1 MiB) as a shortcut; any positive decimal byte count works (e.g. 4096). Each event labeled by source: arena_big_chunk, malloc_big, realloc_big, calloc_big, mmap_big. Note: only fires on the fresh-malloc path; chunks recycled from the per-thread pool bypass this hook — use LOTUS_ARENA_LOG_CHUNK_ATTACH for the full picture.
LOTUS_ARENA_LOG_CHUNK_ATTACH=<N> off Logs every chunk attachment to ANY arena — both fresh-malloc (chunk_attach_malloc) AND per-thread-pool-recycled (chunk_attach_pool) paths — when cap >= N. Use 1 for “log every chunk attachment”. Each event additionally prints arena=<ptr> kind=<root|sub> label=<resolved>: root means the chunk attached to a top-level locus-lifetime arena (a leak class if it grows); sub means a subregion (method scratch / free-fn body) that will recycle to the pool on destroy. label walks the subregion→root chain and looks up the root in the residency registry — requires LOTUS_ARENA_RESIDENCY=1 to populate the label map. Filter kind=root label=<name> to isolate the actual arena-growers. Shares the LOTUS_ARENA_LOG_BIG_MAX_EVENTS cap with the big-chunks logger.
LOTUS_ARENA_LOG_BIG_MAX_EVENTS=<N> 200 Caps the log at <N> events per process. Default 200. Set to 0 for unlimited (useful when watching low-rate sub-MiB allocation patterns over a long window).
LOTUS_CHUNK_POOL_STATS=1 off Dumps per-thread chunk-pool hit / miss / store / overflow counters to stderr at process exit. Diagnostic for “pool isn’t recycling” symptoms — pairs hits vs misses, stores vs overflows. The atexit handler runs on the main thread; counters are __thread so the dump is that thread’s view.
LOTUS_GLIBC_ARENA_MAX=<N> unset (glibc’s own) Calls mallopt(M_ARENA_MAX, <N>) at startup. Caps glibc’s per-thread malloc arena count. 1 forces a single arena (max contention, min virtual-address fragmentation); higher <N> trades contention for parallelism. Useful belt-and-suspenders against the per-thread arena heap-segment proliferation glibc default tuning can produce on long-running daemons. Unset keeps glibc’s default.
LOTUS_BUS_PAYLOAD_ARENA_CAP=<N> 64 MiB Overrides the lazy-global bus payload arena’s byte cap (default 64 MiB). When the cap fires, lotus_arena_alloc returns NULL and the existing alloc-fail paths (empty_global / alloc_failed violation) surface degraded service rather than OOM-killing the process.
LOTUS_DRAIN_GRACE_MS=<ms> 5000 How long a SIGINT / SIGTERM drain may take before the runtime re-raises the signal with its default action, ending the process by it (default 5000). Only in a program that reads self.draining; see semantics.md § “Drain cascade (whole-process)” (GH #1039).
LOTUS_BUS_CALL_ARENA_STATS=1 off At exit, prints [bus call arenas] opened=N live_bytes=L peak_bytes=P to stderr: how many per-call arenas adapter sends and keyed adapter deliveries opened, the chunk bytes still held (0 unless a call is in flight), and the most held at once. The call arenas are not residency targets, so this is how a test sees them (GH #1038).
LOTUS_ARENA_RESIDENCY=1 off Registers every top-level arena (locus __arenas, g_bus_payload_arena, the program-wide global) into a side-table at creation time with a 24-frame construction backtrace. std::process::dump_arena_residency() walks the live set and emits one line per arena to stderr — bytes / chunks / parent / label, sorted by bytes desc — with the construction backtrace. Subregions (method scratch) are skipped; they destroy at method exit and don’t accumulate residency. Atexit also dumps, but post-dissolve fires after all loci tear down — useful only for the global arena’s final state. Long-running daemons should call dump_arena_residency from a heartbeat / checkpoint tick so locus arenas are sampled while still alive.
LOTUS_CHUNK_POOL_PREFILL=<N> 32 Per-thread chunk-pool pre-fill on first touch. Default 32 (= 2 MiB resident per scheduler thread). Set 0 to disable. Bumps the pool’s steady-state floor so brief bursts don’t drain to zero and miss into malloc; the trade-off is per-thread resident memory.
LOTUS_TSAN=1 off (build time) Read at build time (by hale build, not at runtime; see Build-time environment). When set, the emitted clang command passes -fsanitize=thread for both the C runtime compile and the binary link, and skips the -Wl,--wrap=malloc/realloc/calloc/mmap shim surface (TSAN intercepts malloc itself; the wrap’d LOTUS_ARENA_LOG_BIG_CHUNKS diagnostic is silently no-op under TSAN). The resulting binary runs ~5-15× slower; use only for race-hunting workloads. The C runtime embeds an empty __tsan_default_suppressions hook at link time so no external suppression file is needed; all originally-flagged substrate races (bus queue drain, arena destroy, coop pool worker, env-var lazy-init) have been fixed and the suppression list is empty. Opt-in tests live behind #[ignore] and the env var (see crates/hale-codegen/tests/form_hashmap_lockfree_tsan.rs).
LOTUS_LTO=1 off (build time) Read at build time. Opt-in full-LTO native build: the Hale module is emitted as bitcode and the lotus runtime TUs compile with -flto, so the clang -flto -O3 -fuse-ld=lld link inlines the runtime hot paths (arena / string / shm_ring) across the TU boundary into the Hale callers. A few percent on allocation/coordination-heavy code, neutral on vectorized loops (host tuning preserved via per-function target-features). Off by default — ~3–4× slower link, requires lld. Native non-sanitizer only; --wrap shims survive (lld resolves them before LTO codegen). See Native codegen defaults above.
LOTUS_BUS_LOG_UNMATCHED=1 off Surfaces silent no-key-match drops in lotus_bus_local_dispatch_keyed (Phase 3 routing keys). When set, each publish that matches no where key == ... subscriber for the topic emits a single stderr line citing subject, key, and the per-topic subscriber counts (specific vs unkeyed). Off by default — the silent-drop is correct for on_unmatched: swallow topics in steady state, but during bring-up the lack of any signal is load-bearing on debug cycles. Implied by LOTUS_BUS_LOG_DROP=1.
LOTUS_BUS_LOG_DESERIALIZE_DROP=1 off Surfaces silent drops in the udp:// reader thread when (a) no deserializer is registered for the inbound subject, or (b) the deserializer returns <= 0 (size mismatch, bounded-read failure). Emits one stderr line per drop naming the subject, the payload size, and (when applicable) the deserializer’s return value. Off by default; the silent-skip on cross-routed multicast noise is the correct steady-state behavior. Same env-gated pattern as LOTUS_BUS_LOG_UNMATCHED for keyed-dispatch misses. Implied by LOTUS_BUS_LOG_DROP=1.
LOTUS_BUS_QUEUE_CAP=<N> 8192 Caps the cooperative bus dispatch queue, each per-pinned-locus mailbox, and each cooperative pool’s queue at N cells (default 8192; floor 64; rounded up to a power of two; read once). v0.9.0 footprint change: the pinned mailbox and cooperative-pool queues are now lock-free MPSC rings (Vyukov bounded ring + signal-only-when-parked wake), and a fixed-size lock-free ring pre-allocates its cap up front rather than growing to it — so each pinned subscriber mailbox and each cooperative pool now costs ~4.3 MB resident at the default cap (vs the prior grow-as-needed). With the typical handful of pinned loci / pools this is a few-to-low-tens of MB; lower LOTUS_BUS_QUEUE_CAP for pinned-/pool-heavy programs to shrink it (the rings honor it identically). When a producer hits the cap it back-pressures instead of growing without bound (GH #125) — every message is still delivered. The mechanism: a single-threaded producer on the cooperative queue inline-drains it to free space; a cross-thread producer to a full ring blocks (a fenced producers-waiting handshake) until the single consumer drains a slot; a handler self-publishing to its own full ring spills to a consumer-thread-local overflow list (it can’t block on itself). The cross-cooperative-pool shared queue path (multiple drainers, no single consumer) is the remaining non-lock-free path — a follow-on. Lower the cap to tighten the bound / footprint; raise it to reduce drain bursts at the cost of resident memory.
LOTUS_UNIX_STREAM=1 off (SOCK_SEQPACKET on Linux) GH #231: forces the unix bus transport into framed SOCK_STREAM mode on Linux (the Darwin default — macOS has no AF_UNIX SOCK_SEQPACKET). Wire format per message: [u64 LE payload len][u64 LE seq][payload]; the seq is per-connection monotonic from 1, reset per accepted peer, and the receiver counts gaps (seq_gaps counter — GH #236’s loss-computability primitive). Set for EVERY process on a socket: framed and SEQPACKET ends don’t interoperate (a mismatch trips the 8 MB length sanity cap with a diagnostic naming the likely cause). Primary use: Linux CI/test coverage of the macOS code path.
LOTUS_BUS_COUNTERS_DUMP=1 off GH #236 (observability groundwork): prints one stderr line per remote binding at teardown — [bus counters] subject=... kind=... role=... sent= delivered= bytes_sent= bytes_delivered= send_failures= dropped_lost= waits= rearms= reconnects=. The counters are plain relaxed atomics bumped at the transport choke points (fanout send, serve-loop dispatch, re-arm, reconnect) and exist as the substrate for the iris observer; the dump is the operator/test surface until an in-process consumer ships. dropped_lost counts publishes made while a connect binding was in the lost/reconnecting window (GH #233 — drops the publish contract makes deliberate and visible). waits counts publishes that parked in or wait through that window instead (GH #255 phase 1). buffered_early / dropped_early (GH #468) count boot-window wire messages buffered before any matching subscriber registration existed, and the subset dropped (buffer eviction at the 64-msg/1 MiB per-binding cap, a deserialize failure at flush, or a subject that never gained a subscriber by teardown). Socket-buffer occupancy is intentionally not a counter: it’s a poll-time SIOCOUTQ query against the live fd.
LOTUS_BUS_LOG_DROP=1 off Broad superset for diagnosing “publish appears to succeed but handler doesn’t fire” symptoms. Implies LOTUS_BUS_LOG_UNMATCHED + LOTUS_BUS_LOG_DESERIALIZE_DROP AND covers additional silent-drop sites the narrower vars miss: lotus_bus_dispatch’s serialize-fn-returns-<=0 case, the local-fanout (lotus_bus_dispatch_wire + lotus_bus_local_dispatch) zero-matching-subscribers case, per-entry deserialize-returns-<=0 on the local-fanout path, and the no-post-target case (mailbox / coop_pool / global queue all NULL on a matched entry). Each line names the call site, subject, and relevant size / index info so a bus-heavy repro can identify exactly which silent-skip is firing. Reach for this first when investigating bus-drop friction; the narrower vars stay supported for their specific bring-up scenarios.
LOTUS_BUS_QUIESCE_MS 500 GH #468: bound on the main-exit ingress quiesce (default 500, 0 disables it). At every main-exit point, before pools join and loci dissolve, LISTEN binding fds half-close and their readers drain kernel-accepted data to true EOF through the still-intact registry; this bounds how long exit waits for a wedged reader. Exceeding the bound is loud (stderr names it) and drops whatever stayed undrained. Test-only siblings LOTUS_BUS_TEST_BOOT_HOLD_MS / LOTUS_BUS_TEST_READER_STALL_MS deterministically stretch the boot-registration window / the reader’s descheduled window (the loss canaries in binding_ingest_468.rs use them); never set them in production.
LOTUS_ARENA_CHUNK_BYTES_OVERRIDE=<N> 64 KiB Overrides the arena chunk size (F.32-3): a power of two in [4 KiB, 16 MiB], anything else ignored; read once. Smaller chunks keep each locus’s hot chunk in cache across pool rotations in a multi-locus-per-pool deployment. Overridden chunks bypass the per-thread chunk pool.
LOTUS_NO_CHUNK_POOL=<x> off (on under an ASan build) Any non-empty value not starting with 0 turns the per-thread chunk pool off, so a destroyed arena’s chunks are freed instead of recycled with their bytes intact and a use-after-free reads memory a sanitizer can see (GH #816).
LOTUS_HUGE_PAGES=1 off F.32-4a: chunks of at least 2 MiB are mapped with MAP_HUGETLB | MAP_HUGE_2MB instead of malloc’d. A value starting with 1, t or T turns it on.
LOTUS_LOCK_MEMORY=1 off F.32-4c: mlockall(MCL_CURRENT | MCL_FUTURE) at startup (Linux), to remove page-fault stalls on hot-path arena allocation. Needs RLIMIT_MEMLOCK or root; on failure it prints a diagnostic on stderr and runs unlocked. A value starting with 1, t or T turns it on.
LOTUS_BUS_CONFIG=<path> unset The bus config file (the peer and route table hale node writes) that lotus_bus_load_config reads at startup. Unset or unreadable, the program is single-process.
LOTUS_BUS_UDP_RCVBUF=<N> the kernel’s SO_RCVBUF, in bytes, for the udp bus readers. Ignored unless a positive int.
LOTUS_BUS_TEST_BOOT_HOLD_MS=<ms> 0 Test only: stretches the boot-registration window of a listening binding (see LOTUS_BUS_QUIESCE_MS). Never set in production.
LOTUS_BUS_TEST_READER_STALL_MS=<ms> 0 Test only: stretches the window in which a binding’s reader is descheduled. Never set in production.
LOTUS_TEST_PINNED_START_NO_DRAIN=1 off Test only: disables the instantiating thread’s queue drain while it waits for pinned or pool initialization. The startup request/reply negative controls use it to expose the resulting deadlock. Any non-empty value not starting with 0 enables it. Never set in production.
LOTUS_LIFECYCLE_SKIP=<steps> unset Test only, and read only by a lifecycle-trace build (HALE_LIFECYCLE_TRACE=1; a release runtime has no such code): a comma list of steps a negative control removes, a kind (PoolJoin) or one event line (Reclaim.Completed). See The lifecycle trace.
HALE_MATRIX=full the sample Test only, read by the test suite (ownership_matrix.rs, lifecycle_matrix.rs), never by a program: full runs every cell of the generated ownership and lifecycle matrices instead of the default deterministic sample.
LOTUS_OBS=1 off Native observation emission (iris): the process creates its observation segment and its probes emit. Implied by LOTUS_OBS_RECORD and LOTUS_REPLAY. See Native observation emission.
LOTUS_OBS_RINGS=<N> 8 (64 when recording) Rings in the observation segment, 1 to 64; a value outside that falls back to the default.
LOTUS_OBS_SLOTS=<N> 4096 Slots per ring: a power of two, at least 64; anything else falls back to the default.
LOTUS_OBS_WIRE=1 off Puts the (origin, seq) edge identity on the wire: the udp magic prefix and the framed transport’s origin widening. That is a wire-format change a pre-header receiver cannot parse, so the whole fleet opts in together; it is what cross-process edges need (iris handoff-4 P16).
LOTUS_OBS_RECORD=<path> unset Lossless recording (GH #296): write the run’s tape to <path>; implies observation. See Lossless recording mode.
LOTUS_OBS_RECORD_DURABLE=1 off Synchronize every flushed drain sweep of the recording (fdatasync on Linux, F_FULLFSYNC on macOS); the default rides the page cache (survives a process crash, not power loss).
LOTUS_OBS_RECORD_ENV=full withheld Record the value of every environment read, not just its name, existence and length; the artifact header says which policy applied.
LOTUS_REPLAY=<path> unset Replay a recording (hale replay sets it): the run is judged against the tape.
LOTUS_REPLAY_FEED=<path> unset Feed mode (hale replay --feed): the recording is input, not law. Mutually exclusive with LOTUS_REPLAY.
LOTUS_REPLAY_ALLOW_TRUNCATED=1 off Accept a recording with no clean-finalize trailer (a crashed writer); --allow-truncated.
LOTUS_REPLAY_ALLOW_UNVERIFIED=1 off Replay although the recording’s model identity differs from this build’s; --allow-unverified.
LOTUS_REPLAY_FEED_ALLOW_UNMATCHED=1 off In feed mode, unfed tape is a failure unless this is set.
LOTUS_REPLAY_STATUS=<path> unset Where the replayed process writes the verdict counters (journal misses, order divergences) the CLI reads.
LOTUS_REPLAY_AT=<n> unset Stop the replayed run (SIGSTOP) at the n’th consume; hale replay --at <n>.
LOTUS_REPLAY_AT_CONSUMER=<id> unset With LOTUS_REPLAY_AT, count the n’th consume of that consumer instead of the process-wide n’th; --at <consumer-id>:<n>.
LOTUS_REPLAY_FD=<fd> unset An already-open, validated descriptor for the recording, passed by hale replay so no path is re-resolved between admission and replay.
LOTUS_API_ROLES=<table> the table param of the program’s std::api::StaticRoles Overrides a StaticRoles source’s table at run time (spec/stdlib.md § std::api); with no table, every requires refuses until this says otherwise.
HALE_LOG=<level> unset: no filtering std::log drops events below this level (debug, info, warn, error, case-insensitive). Anything unrecognized, and an unset variable, filters nothing, so a typo cannot discard the logs it was set to see.
HALE_VAULT_DIR=<dir> $XDG_CACHE_HOME/hale/vault, else ~/.cache/hale/vault, else /tmp/hale-vault The local vault directory a vault: source of std::secret reads (a file per name).
HALE_VAULT_ADDR=<url> unset When set, a vault: source resolves over the vault’s HTTP API at <url>/v1/secret/<name> instead of the local directory.
HALE_VAULT_TOKEN=<token> unset The bearer token for HALE_VAULT_ADDR; empty, the lookup fails closed.

Every top-level arena is created via lotus_arena_create_labeled(name) and carries an immutable human-readable label string. The codegen passes the locus name (e.g. WsClient, __lib_metrics_metrics_MetricMap); the program-wide global is labeled lotus.arena.global; g_bus_payload_arena labels itself. The label is the load-bearing identifier in the residency dump; backtraces resolve via -rdynamic for cases where the label alone isn’t enough.

The arena-chunk pool and -wrap=malloc family ship in every binary unconditionally; the env vars are zero-cost when unset (one int read + one branch per allocation). The -rdynamic link flag is similarly unconditional so backtrace symbols resolve without addr2line.

Runtime size budget

Section titled “Runtime size budget”

The runtime should be small enough that a hello-world program binary is < 1 MB statically linked, and < 100 KB if dynamic linking against libc. This is a target, not a guarantee.

The framework’s discipline enables this: no GC, no metadata overhead per allocation, region-based MM compiles to bump allocators. Comparable to C in size, with ergonomics closer to Erlang.

Open questions for runtime

Section titled “Open questions for runtime”