Runtime

Reference material, synced from the compiler repo’s spec/. The guide is the gentler path in.

What every compiled Hale binary always ships with. Always- loaded; not optional; no import needed; the substrate every Hale program depends on.

This document distinguishes the runtime (always there) from the standard library (stdlib.md, importable but bundled). Go’s distinction between runtime and other stdlib packages is the model: runtime is automatic; stdlib is explicit.

Naming note: The language is Hale; the runtime/ substrate concept is called lotus, and the C-runtime symbols stay lotus_* (per project memory). When this doc says “lotus” it means the substrate; “Hale” means the language proper.

What’s in the runtime

Section titled “What’s in the runtime”

Memory

Section titled “Memory”

Lifecycle

Section titled “Lifecycle”

Scheduler — multi-scheduler cooperative

Section titled “Scheduler — multi-scheduler cooperative”

Lotus uses a multi-scheduler cooperative model (closest existing analog: Erlang BEAM, not Go’s M:N). The reasons are framework-discipline:

Concurrency comes from running multiple cooperative schedulers in parallel (one per CPU core, by default). Loci belong to a specific scheduler; cross-scheduler communication uses the bus just like cross-process communication. Loci may be migrated between schedulers transparently for load balancing because all their communication is bus-mediated already.

Specifically:

Placement classes (per-locus execution strategy)

Section titled “Placement classes (per-locus execution strategy)”

Just as projection class governs a locus’s memory strategy, placement class governs its execution strategy. Placement is a deployment seam, not an intrinsic property of the locus (see spec/decisions.md § F.31). Placement entries live in a placement { } block on main locus only, parallel to bindings { } for bus topology:

main locus App {
params {
gateway_a: Gateway = Gateway { venue: "venue-a" };
gateway_b: Gateway = Gateway { venue: "venue-b" };
metrics: MetricsServer = MetricsServer { port: 9100 };
ui: Renderer = Renderer { };
}
placement {
gateway_a: pinned(core = 1);
gateway_b: pinned(core = 2);
metrics: cooperative(pool = io);
ui: cooperative(pool = render);
// unspecified main-locus params → cooperative(pool = main)
}
}

Placement remains honestly bimodal: either a locus shares a cooperative pool (an OS thread running a cooperative drain loop) or it owns its own OS thread. There is no third position.

Class Yield discipline Resource
cooperative(pool = X) (default for unspecified main-locus params, with X = main) Yields between substrate cells (handler exit, lifecycle transition, bus dispatch, time::sleep, explicit yield). time::sleep slices into ≤100ms intervals and folds in lotus_bus_queue_drain for the locus’s pool after each slice, so cells posted by other threads deliver mid-loop even during a long keep-alive sleep. Handler bodies are atomic. Shares pool X’s OS thread with other cooperative loci placed on the same pool.
pinned / pinned(core = N) / pinned(cores = A..B | A..=B | {a, b, c}) / pinned(node = N) / pinned(l3 = name) / pinned(..., replicas = K) No yield to siblings; owns its OS thread. Bus events to/from cross-thread boundaries via formal mailbox post. Dedicated OS thread. core = N pins it to one CPU; cores = ... (Phase 1a) sets its mask to a core set; node = N / l3 = name (Phase 1b) set the mask to a NUMA node / cache domain from topology { } (and bind the arena there); replicas = K (Phase 1c) fans into K single-threaded instances, one per core. The OS schedules freely within the mask. Linux-only; best-effort no-op elsewhere.

Pool inference rule. The cooperative pool set is inferred from cooperative(pool = X) references in the placement { } block. The runtime spawns one OS worker thread per inferred pool name beyond main (which is always the program’s main thread). No separate threads { } declaration block at v1 — when per-pool attributes (priority, affinity, realtime hint) become useful, the block lands then as a typed extension. Pool main exists in every program regardless of whether placement { } references it.

Nested-instantiation inheritance. Placement entries apply only to top-level main locus params fields. Loci instantiated nested in another locus’s body (in birth / run / lifecycle methods, or as let-bound children) inherit the parent’s pool. There is no way to spell “this nested child runs on a different pool than its parent” — that would require the nested-instantiation expression to carry placement, which would re-mix the deployment and intrinsic layers F.31 separates.

Topology block (Phase 1b). A main locus may also declare a topology { } block — a declare-only description of the host’s core partition, a sibling deployment seam to placement { } / bindings { }:

topology {
reserve cores 0..2; // held back for the OS / main
node 0 {
l3 fast { cores 4..8; } // a CCD / shared-L3 group
l3 slow { cores 8..12; }
}
node 1 {
l3 heavy { cores 12..16; }
}
}

A placement { } entry then targets a domain: pinned(node = 0) sets the thread’s affinity mask to node 0’s core set (the union of its L3 domains — here {4..12}), and pinned(l3 = fast) sets it to the named domain’s cores ({4..8}). The compiler resolves the domain to a concrete core set at compile time (closed-world: node ids and domain cores are literals), reusing the same cpuset affinity mechanism as pinned(cores = ...). Validation (unique node ids, globally-unique L3 names, non-overlapping domains, no domain/reserved overlap, and every pinned(node/l3) referencing a declared domain) is static. L3 domain names go through the identifier rule, so a hard keyword (e.g. bulk) can’t name a domain.

Replicas (Phase 1c). pinned(..., replicas = K) is the parallelism sugar: it fans the field into K single-threaded instances, replica i pinned to one core of the affinity set (round-robin — pinned(cores = 4..12, replicas = 8) puts replica i on core 4 + i; more replicas than cores wraps; with no affinity the K instances are OS-scheduled). This is deliberately not a multi-worker pool — a cooperative pool is one consumer thread, and the lock-free rings, bus devirtualization, and single-threaded-method guarantee all rest on that invariant. Parallelism instead comes from more single-threaded units, each its own single consumer, so every invariant survives. replicas is pinned-only (K cooperative loci on one pool would share a thread, which isn’t parallel) and composes with the topology targets (pinned(node = 0, replicas = 4) fans across node 0’s cores with each replica’s arena bound to node 0). Codegen emits K instantiations at the field’s init site; all K register their bus subscriptions (a subscribed topic fans out to every replica) and all K are joined + dissolved at parent teardown via the deferred- dissolve frame. The replicas are non-addressable — there is no field[i] surface; they are workers that pull from the bus or run their own loop.

Thread + memory co-location. A pinned(node = N) / pinned(l3 = name) locus binds not just its thread but its memory: its arena is created via lotus_arena_create_labeled_on_node, which flags the arena’s NUMA node, and every chunk that arena grows is mmap’d (page-aligned) and bound to the node with the mbind syscall (MPOL_BIND) before first touch — so pages fault in on the node regardless of which thread touches them first (the locus struct is instantiated on main but runs on its own pinned thread). Sub-regions inherit the node, so a node-pinned locus’s method scratch — the dominant per-invocation allocation — lands on its node too. mbind is invoked as a raw syscall, so this adds no libnuma dependency; the whole feature is zero-cost for programs that don’t opt in (an unbound arena, the default, takes the ordinary malloc / chunk-pool path byte-for-byte). Best-effort and Linux-only, exactly like pinned(core = N): an mbind the box can’t honor (node absent, capability denied) falls back to first-touch, and on non-Linux hosts the arena allocates normally. (Huge-page-backed chunks and node binding don’t currently combine — a node-bound arena uses regular pages; a follow-up.)

Single-threaded-method invariant. A locus’s methods may be invoked only on the OS thread that owns its placement’s pool. Cross-pool method calls and lateral field accesses go through the bus’s existing copy-and-condvar dispatch machinery, which already crosses thread boundaries safely. The typechecker walks the static call graph from each top-level placement entry, propagates pool ownership through method receivers, and rejects calls that cross pools without going through the bus. This is the substrate enforcement that makes M:N safe — without it, multi-pool deployments would silently race on locus arenas (which are unsynchronized bump allocators).

The single shared bus payload arena is the deliberate exception to “one owning thread per arena”: it is reachable concurrently from any pool, because some stdlib primitives allocate their result there directly rather than into a per-locus arena (e.g. std::io::tls::recv_bytes, which always targets it regardless of the caller’s per-thread scratch). That arena therefore carries an internal lock on its bump — two pinned loci calling such a primitive at once do not corrupt each other’s allocations. Per-locus arenas keep the lock-free bump; only this one shared arena pays for the lock.

Why no “greedy” class

Section titled “Why no “greedy” class”

A natural temptation is to want a third option: “shares a pool’s thread but doesn’t yield.” That would be a bimodality violation. Cooperative already guarantees handler-level atomicity — no preemption within a substrate cell — so the only thing such a class could add over cooperative is “don’t yield between cells either.” But that means leaving the shared pool entirely. The place you go when you leave is your own thread. That’s pinned.

Latency-critical work, or anything that genuinely shouldn’t share with neighbors on its pool, is signaling that it belongs on its own pool (placed cooperative(pool = some_quiet_pool)) or on its own thread (placed pinned). The first option is new in F.31: pools partition the cooperative substrate so “shouldn’t share with siblings” no longer forces pinned — but the underlying bimodality (cooperative-pool vs pinned-thread) holds.

time::sleep drain semantics

Section titled “time::sleep drain semantics”

The codegen lowering of std::time::sleep(d) slices the request into intervals of at most 100ms and ends each slice’s EINTR-retry loop with an inline call to lotus_bus_queue_drain against the program-wide cooperative queue (plus a pinned-mailbox drain). The total wall-clock sleep is preserved (the slices sum to d); sleeps of 100ms or less take exactly one slice, so the common case is unchanged. A cooperative subscriber looping

run() {
while !self.bail {
std::time::sleep(100ms);
// ... loop body sees handlers fired during the sleep
}
}

receives cells posted by other threads — unix-bound reader threads, pinned publishers via lotus_bus_local_dispatch, etc. — right when each sleep returns, without an explicit yield;. The drain is idempotent, so existing code with sleep; yield; stays correct.

Which path delivers to whom. The queue drained here is the program-wide in-process cooperative queue, drained on the main thread — so this sleep-loop drain delivers in-process cells to a cooperative subscriber on the main pool. It is not the only delivery path: cross-process topics (udp:// / unix:// bound via LOTUS_BUS_CONFIG) are delivered by the transport reader thread, which dispatches directly into the subscribed handler set and reaches a cooperative locus on any pool — provided that pool’s thread is free to run the dispatch. So a non-main cooperative subscriber receives reliably as long as it doesn’t monopolize its pool thread with a blocking call. (Pinned loci are a third path: a per-locus mailbox drained at each sleep/yield.) The failure mode is a cooperative subscriber whose run() blocks — the dispatch can’t run and its handlers never fire; that blocking-and-subscribing combination is what the dead-bus-receiver rule rejects (see “Type-check rules” in spec/semantics.md), not non-main placement on its own.

Long sleeps no longer starve main-pool handlers. Before the slicing, the drain happened only after the whole sleep returned, so the natural keep-alive idiom

run() { while true { std::time::sleep(60s); } } // on the main thread

starved every pool = main bus handler for 60s at a time: a main-pool subscriber registers with coop_pool == NULL, so the wire/reader-thread dispatch path lands its cells on the global cooperative queue (g_bus_queue_for_remote, the same object only main drains). A 60s blocking sleep on main left those cells unserviced for the full 60s — indistinguishable from “the handler never fires.” Slicing keeps the queue serviced ~10×/s during any sleep, so a main-pool handler that republishes onto a topic an async-pool subscriber listens to (e.g. a udp-reader-fed producer forwarding to per-connection writers) now flows promptly regardless of how long main is asleep.

The pinned-mailbox path is unchanged: pinned subscribers wake on lotus_mailbox_post’s condvar broadcast regardless of what the cooperative scheduler is doing.

Owner-executed handlers (2026-07-15, downstream handoff)

Section titled “Owner-executed handlers (2026-07-15, downstream handoff)”

Two runtime rules restore the single-threaded-locus invariant (F.31) dynamically — the compiler already enforces it for direct calls, but two bus paths used to violate it under load (reproduced as a SIGSEGV in a 10k msg/s ingest bench):

  1. The global cooperative queue is drained only by its owner thread (main, recorded at queue creation). The scope-exit flush emitted at the end of every fn/method body — and the sleep-slice / yield drains — call lotus_bus_queue_drain on whatever thread ran the body; on any thread but the owner the call is now a no-op. Previously a pinned publisher’s flush would execute a main-pool subscriber’s handler on the publisher’s thread, concurrently with main’s own drains (the locked drain releases the queue mutex before each handler invocation) — two threads inside one locus.
  2. Payload deserialization happens on the subscriber’s owner thread. The non-flat dispatch paths deserialize each published payload into the subscriber’s arena (Task-11 arena routing); for a target owned by a different thread this write used to happen on the publisher’s thread — an unlocked cross-thread write into a foreign arena. Now a cross-thread publish enqueues the wire bytes plus the deserialize fn in the cell, and the owner materializes (deserializes into its own arena) at drain, just before invoking the handler. Same-thread targets keep the deserialize-at-dispatch fast path, so single-pool programs are unchanged.

Consequences: a main-pool subscriber’s handlers run only on main (at sleep-slice / yield / scope-exit drains — worst-case ~100ms after a cross-thread publish, per the slicing above); pool subscribers run only on their pool’s worker; pinned subscribers only on their own thread. Flat (pointer-free POD) payloads are exempt from rule 2 — a verbatim byte copy writes no arena. Delivery and FIFO order per subscriber are unchanged.

Item B from the 2026-05-21 friction log (a cooperative publisher’s <- to a pinned subscriber) is resolved as of 2026-06-01 — the earlier “drains only at dissolve in some configurations” was a sequencing effect, not a lost wakeup. The mailbox condvar path is correct: a pinned subscriber whose run() returns proceeds into the blocking lotus_mailbox_drain_one, and a cooperative publisher’s <- wakes it via the not_empty broadcast — confirmed by coop_to_pinned_mid_program::pinned_returning_run_drains_mailbox.

The residual constraint is inherent to a single pinned thread: a pinned subscriber with a long-running run() that never returns or yields cannot drain its mailbox during run() — the one thread is busy in the loop. Such a run() must reach a cooperative yield point (time::sleep / yield) for the TLS-cached lotus_mailbox_drain_pending to service the mailbox (the coop_to_pinned_mid_program sleep-loop test), or let run() return so the post-run blocking drain takes over. This is a property of the bimodal model — “pinned owns its thread, no yielding between cells” — not a bug; a pinned subscriber that must receive bus traffic while busy should yield in its loop. Cross-binary flows (unix / shm_ring with their own reader threads) land in g_bus_queue and benefit from the sleep-folded drain above.

Long-running cooperative children: placement closes Item D

Section titled “Long-running cooperative children: placement closes Item D”

Pre-F.31, declaring a long-running cooperative child as a params field (metrics_server: std::http::Server = ...) serialized parent and child onto the main thread: the child’s run() body never returned, so the parent’s run() never started. The workaround was the sibling-in-main pattern — hoisting the child to a top-level main locus param so it ran “alongside” the parent rather than “inside” it.

Under F.31 the sibling-in-main pattern IS the canonical shape, and it composes cleanly because placement { } lets each sibling pick its own pool:

main locus App {
params {
gateway: Gateway = Gateway { };
metrics: std::http::Server = std::http::Server { port: 9100 };
}
placement {
gateway: pinned(core = 1);
metrics: cooperative(pool = io);
}
}

Parent’s run() no longer serializes against metrics’s accept loop — they sit on different OS threads. To shut down gracefully when one finishes (e.g. a duration-bounded gateway exits), call metrics.shutdown() from the finishing locus’s thread; the C-iii interruptible-accept work makes this the supported pattern.

The nested-as-child shape (long-running cooperative locus as a params field of a non-main locus) remains structurally serialized — that’s a consequence of the nested-instantiation inheritance rule (children share their parent’s pool by construction). Nested long-running children are an antipattern under F.31: hoist to main-locus siblings.

Typecheck enforcement. The compiler rejects the antipattern at typecheck. A non-main locus with a non-trivial run() body holding a params field of a locus type whose own run() is also non-trivial — including std::http::Server and the other entries on the known-long-running stdlib allowlist — gets a hard error pointing at the canonical sibling-in-main + placement fix. The runtime starvation that motivated this rule is silent (the parent’s run() simply never executes), so the type-side rejection is load-bearing: it converts a class of hard-to-diagnose runtime bugs into a clear compile-time signal.

where async_io — green-I/O cooperative pools (F.35)

Section titled “where async_io — green-I/O cooperative pools (F.35)”

The sibling-in-main fix puts each long-running child on its own OS thread, which caps concurrent connections at one-per-pool. To scale beyond that without spawning a thread per connection, a placement entry may declare where async_io:

main locus App {
params {
listener: std::websocket::Server = std::websocket::Server { ... };
worker: WsWorker = WsWorker { ... };
}
placement {
listener: cooperative(pool = ws_accept) where async_io;
worker: cooperative(pool = ws_workers) where async_io;
}
}

where async_io opts the pool into green-I/O scheduling: the pool’s worker drain loop integrates an epoll instance, and blocking I/O syscalls inside locus methods on this pool park- and-resume instead of blocking the OS thread. The user code inside the locus is unchanged — recv_bytes(stream) reads the same line of source whether the pool is async_io or not; the substrate picks the right lowering at the syscall boundary.

One worker thread per named cooperative pool — a promise, not an implementation detail (Crumb batch-3, 2026-07-28). Every named cooperative pool (async_io or classic) has exactly one OS worker thread for the lifetime of the program. Consequences a consumer may depend on: every run(), bus handler, and parked- coro resume for loci on the pool executes on that one thread (coroutines interleave on it; they never migrate), so a thread-affine C library — a JS engine, SQLite in serialized mode, a GUI toolkit — placed on a named pool is entered from a single thread by construction, and @export re-entry from foreign code called on that thread stays on it. Concurrency within a pool is cooperative only. If a scale-out mechanism ever lands, it will be a new opt-in placement form; cooperative(pool = name) keeps the single-worker guarantee.

Because parking yields the shared worker, N reader loci that each park on their own fd — the F.35 one-reader-per-signal shape — are serviced concurrently by a single pool. Two invariants make that multiplexing correct: (1) the drain loop starts a queued run() the moment the running coro parks, not only when it completes, so a long-lived reader never starves the readers queued behind it; and (2) each coro’s caller-arena — the thread-local that decides where its stdlib allocations (recv result blobs, string builders) land — is snapshotted across the park and restored on resume, so a coro that resumes after a sibling ran (and perhaps dissolved) never allocates through an arena the sibling has since torn down. Every blocking-recv primitive on the pool honors invariant (1) by parking rather than blocking recvfrom/read (the std::io::udp recv family joined the tcp/tls siblings here in the 2026-07-15 downstream handoff, and std::time::sleep joined them 2026-07-28 (Crumb batch-5) via a timer-only park — a deadline with no fd, serviced by the same expiry sweep — so sleeping coros overlap on one worker and a sleep inside a handler never blocks the pool; the timer half of an event loop now costs what the design always claimed).

Each bus delivery to a subscriber on an async_io pool runs its handler on a coroutine (a struct + a 64 KiB stack). Rather than allocate and free that pair per delivery, the pool keeps a bounded per-worker free-list (cap 64) of completed coro slots and reuses them — a warm fan-out skips the per-dispatch stack allocation entirely (2026-07-16). The free-list is worker-thread-local (no lock) and drained at pool teardown, so a busy async pool retains up to 64 × 64 KiB (~4 MiB) of coro stacks at steady state. Transparent to user code — a pure allocation optimization, no behavior change.

Typecheck rules:

See spec/decisions.md § F.35 (forthcoming) for the green-I/O substrate design + perf-axis trade-offs.

(Compare: rich / chunked / recognition projection classes are genuinely three-way because N≈10, N≈30, and N≈300 are different cost regimes at scale — memory has more genuine intermediate ground than time does. Placement, even with M:N pool partitioning, stays bimodal.)

Cross-class bus semantics

Section titled “Cross-class bus semantics”

Implementation status (m26 + m27 + m28a + m28b + m28c; F.31 pending)

Section titled “Implementation status (m26 + m27 + m28a + m28b + m28c; F.31 pending)”

m25 wired the annotation through parse / typecheck / codegen. m26 ships cooperative semantics; m27 ships pinned threads (run-only); m28a lifts pinned to full lifecycle; m28b lights up cross-thread bus mailboxes — pinned loci can subscribe and publish, with cells routed across threads via per-locus mailboxes; m28c adds optional CPU-core affinity via pthread_setaffinity_np.

F.31 (2026-05-23, Phase 1-5 + Phase 4a shipped): the placement-at-main surface and M:N cooperative pools. The per-locus : schedule annotation is removed; the placement choice moves to a placement { } block on main locus. Cooperative subscribers can be partitioned across N pools (N OS threads, each running its own lotus_coop_pool_worker drain loop against its own per-pool ring buffer); cross-pool bus dispatch reuses the m28b condvar+memcpy machinery. A pool’s worker thread can take the same affinity forms as pinnedcooperative(pool = X, core/cores/node/l3 = …), applied right after the worker spawns (2026-08-12; entries naming one pool must agree, and the main pool takes none). The single-threaded-method invariant — a locus’s methods may be invoked only from its pool’s thread or via the bus — ships as a typecheck rule (Phase 5).

Phase 4 v1 limit (handler-only cross-pool delivery). The runtime ships pool-aware bus dispatch for now: a subscriber whose enclosing locus is placed on a non-main cooperative pool gets its handler invoked on that pool’s worker thread. Lifecycle methods (birth / run / dissolve / accept) still run on the main thread for cooperative-pool loci — the codegen does NOT yet relocate them to the pool worker. For state mutated only inside bus handlers this is enough to honor the single-threaded-method invariant (the handler is the only writer of locus state on the pool thread). State touched by both lifecycle bodies and handlers on a non-main pool is a Phase 4b concern; the typechecker doesn’t flag it today because lifecycle methods are not user-callable in the recv.method() shape that Phase 5 checks. Plan: Phase 4b moves lifecycle dispatch onto the pool worker via the same queue mechanism (post “run_init” / “drain_exit” cells at instantiation / scope-exit boundaries).

Runtime pool inheritance for in-method-body instantiation (2026-05-29). A locus instantiated inside a method or bus-handler body that is itself executing on a pool worker inherits that pool at runtime. Codegen has no static placement name for such a locus (placement keys on main-locus params fields only), so the run-posting site and the subscription-registration site resolve the pool as: the compile-time-known pool when the locus IS a placed main-locus field, else the pool whose worker is currently on-CPU (lotus_coop_pool_current(), which reads the per-thread g_current_pool_tls; NULL on the main thread). When the resolved pool is non-NULL the child’s run() is posted to it (its own cell — and, on an async_io pool, its own parkable coro, so a blocking recv in the child’s run() parks that child’s coro rather than the spawning handler’s) and its bus subscriptions are tagged with that pool so dispatch routes to the right worker. This is gated on the child being owned beyond the spawning scopeaccept’d, an owned param field, or returned. A handler-local let-bound long-lived locus is not owned (its deferred dissolve fires at the handler’s scope exit), so posting its run() would execute after it’s dissolved; those keep the prior behavior (synchronous run(), global-queue subscription). The canonical N-dynamic-children shape (per-connection handlers, per-tenant workers) is therefore an accept’d child whose run() holds the recv loop — it multiplexes on an async_io pool by construction. See spec/memory.md § “Owned param-field child allocation” for the companion arena rule that makes the owned child’s full subtree outlive the spawning frame.

Adapter loci instantiated inline in bindings { Topic: AdapterLocus { ... }; } are NOT main-locus params fields and so receive no placement { } entry. Their run() recv- loops need a dedicated thread by construction; the substrate places them pinned-equivalent implicitly (same m90 routing + pthread spawn that pre-F.31 fired via the adapter’s : schedule pinned annotation). The annotation goes away but the behavior is preserved automatically — the bindings-inline shape unambiguously signals “transport adapter with a recv loop.”

The implementation notes below describe pre-F.31 shapes (m25 through m28c). They remain accurate for the cooperative-only- on-main-pool case, which is the v1-compatible default when no placement { } block is declared.

m26 (cooperative): Each <- enqueues (handler, self, payload_copy) cells onto a program-wide FIFO queue (@lotus.bus_queue.global) instead of running handlers inline. The scheduler drain loop pops cells one at a time and invokes the handler — handler-atomic per substrate cell, with cooperative yields BETWEEN cells rather than nested call frames. Handlers may publish more events; drain continues until empty.

Drain runs at the start of every flush_dissolve_frame — before any long-lived locus dissolves — so subscribers process pending cells while still alive. Plus an explicit yield; statement (m26b) drains at user-placed points inside long internal loops. v0 limitation: cells enqueued DURING a dissolve are leaked.

The C runtime gained the queue surface:

ptr lotus_bus_queue_create(void)
void lotus_bus_queue_enqueue(ptr q, ptr handler, ptr self, ptr payload)
void lotus_bus_queue_drain(ptr q)
void lotus_bus_queue_destroy(ptr q)

m20’s “memcpy payload into subscriber’s arena” step happens at ENQUEUE time (publisher’s frame).

m27 + m28a (pinned threads + full lifecycle): Pinned-class loci spawn a pthread at instantiation; the locus’s full declared lifecycle (birth → run → drain → dissolve, each only if declared) executes on that thread, in order. Main thread continues immediately after spawn. At scope exit (deferred- dissolve flush), pthread_join blocks until the pinned thread has finished its lifecycle and returned; the main thread’s only remaining work for a pinned entry is the join plus the locus’s arena destroy wholesale (drain / dissolve are SKIPPED on the main side — they ran on the pinned thread).

m28a synthesizes a per-locus __pinned_main_<LocusName> function whose signature matches pthread’s start-routine contract directly (ptr (ptr)); pthread_create gets that function pointer with self_ptr as its argument. No C-side adapter, no thread_args struct. The synthesized body simply calls each declared lifecycle method in sequence, then returns null.

m28b stage 1 (inline-payload queue): Bus queue cells now carry an inline [u8; 512] payload buffer (with pthread_mutex_t guarding the cell array) instead of a pointer to subscriber-arena memory. The publisher memcpy’s into the cell at enqueue; the drain (running on the subscriber’s thread) memcpy’s from the cell into the subscriber’s arena before invoking the handler. This makes the queue the single point of cross-thread synchronization: each per-locus arena stays single-threaded territory, the boundary between layers is where the lock lives. Per spec/memory.md, “every locus boundary copies the payload” still holds — just with two memcpy’s per cell instead of one.

m28b stage 2 (cross-thread mailboxes): Each pinned locus that declares bus subscribe allocates its own lotus_mailbox_t at instantiation: a bounded ring buffer with pthread_mutex_t + pthread_cond_t + a shutdown flag, sharing the same inline-payload cell shape as the global queue. The locus’s struct grows a __mailbox: ptr field to hold it.

The bus entry table grows from {subject, self, handler} to {subject, self, handler, mailbox}. Cooperative subscribers register with mailbox = NULL; pinned subscribers register with their mailbox pointer. At dispatch time, the bus_dispatch fn loads entry.mailbox and branches: null → enqueue on the global cooperative queue (handler runs on the cooperative thread); non-null → lotus_mailbox_post on the pinned subscriber’s mailbox (handler runs on the pinned thread).

The synthesized __pinned_main_<Locus> body grows a mailbox loop between run() and drain(): it calls lotus_mailbox_drain_one, which blocks on the condvar until either a cell arrives (returns 1, after dispatching the handler) or shutdown is signaled with empty queue (returns 0, breaking the loop). Pending cells flush before the loop returns 0 even after shutdown — the order check is “queue empty AND shutdown.”

Coordinated shutdown: at the deferred-dissolve flush, the main thread calls lotus_mailbox_shutdown on the pinned locus’s mailbox (sets the flag + broadcasts the condvar), then pthread_join. The pinned thread observes the empty+shutdown condition, breaks its loop, runs drain() and dissolve(), and exits — main joins, then destroys the mailbox and the arena.

Per The Design / lotus, this is the canonical “any → pinned” bus path: publisher and subscriber sit in different layers of the lotus, the substrate cost lives at the layer boundary (the mailbox lock + the inline payload’s two memcpy’s), and each arena stays single-threaded territory. Bimodality holds.

Still gated: pinned loci cannot declare accept() (children of pinned would need cross-thread cascade-dissolve coordination, which is meaningful new infrastructure beyond m28b’s mailbox post-and-continue) or closures (cross-thread violation routing). Codegen errors clearly if those are present.

m28c (CPU-core affinity): When a pinned locus declares : schedule pinned(core = N), codegen emits a call to lotus_set_core_affinity(tid, N) immediately after pthread_create succeeds. The C-side helper wraps pthread_setaffinity_np (with a cpu_set_t zeroed and bit N set) so codegen doesn’t have to know the cpu_set_t layout (opaque + size-variable across glibc versions). Best-effort semantics: if the requested core is unavailable (e.g., CI box with fewer cores than the source declares) or the syscall is denied, the runtime silently falls back to ordinary OS scheduling rather than refusing to run the binary. The underlying bimodality is unchanged — pinned(core = N) is a refinement WITHIN the pinned mode, not a third position.

Topology Phase 1a (cpuset affinity): pinned(cores = A..B) / pinned(cores = A..=B) / pinned(cores = {a, b, c}) generalize the single core to a core set: the thread’s affinity mask is the whole set and the OS schedules it freely within it — a range carves out an isolation domain rather than picking one CPU. Bounds are integer literals (placement is a closed-world deployment seam), so the compiler expands the spec statically — sorted, deduplicated — into a constant array and emits one call to lotus_set_core_affinity_set(tid, cores, count) after pthread_create. Range inclusivity follows expression ranges: .. excludes the upper bound, ..= includes it. The typechecker rejects a spec that selects no cores (4..4, 8..=4) and a duplicated set element; whether the cores exist on the deploy box stays best-effort at runtime — the C helper skips out-of-range indices and applies the mask only if at least one valid core remains. CPU affinity is Linux-only: on other hosts (macOS) both helpers are compiled as no-ops and the loci run unpinned. pinned(core = N) continues to route through the single-core helper unchanged.

Linker dependency: clang invocation now passes -lpthread unconditionally; small fixed cost in the resulting binary (libpthread is on every modern Linux).

Bus message router

Section titled “Bus message router”

The runtime’s bus is transport-agnostic. From the framework’s perspective, a transport is the bus kernel projected through a parameter regime: NATS and UDP multicast and TCP and Unix sockets are the same primitive (typed pub-sub) at different (B, c, σ, φ) values. The runtime knows about subjects, channels, and modes; specific transports come from stdlib (std::bus::*).

v1.x source surface. Subjects are now declared as typed top-level topic Foo { payload: T; subject: "..."; } decls (with optional : Parent for hierarchical wire subjects); deployment-time bindings live in the main locus’s bindings { Topic: <transport>; } block. Two transport shapes ship: substrate-provided unix("/path", role: ...) and user-supplied adapter loci named directly on the right-hand side (any locus satisfying __StdBusAdapterfn send(subject: String, bytes: Bytes) — qualifies). In-memory delivery is absence-of-entry. Adapter bindings let protocol-layer transports (NATS, MQTT, TCP-with-framing, custom JSON-over-WebSocket) live in user code without the language having to enumerate protocol variants. See spec/semantics.md “Topic declarations → Phase 2” for the full surface, including the closed-world topology optimization that elides bus dispatch for unambiguous intra-locus and single-hop parent→child tower patterns when no binding is declared.

Binding realization failure (GH #227) + transports-as-loci (GH #233 steps 1–2, 2026-07-22). Source-level bindings { T: unix(...) } entries are sugar: codegen instantiates a stdlib transport locus (__StdBusUnixListenTransport / __StdBusUnixConnectTransport, runtime/stdlib/bus.hl) as a cooperative child at the main prelude — converging with the adapter path. The locus is the control plane; the data plane stays in C:

Publish fanout is untouched — realized entries land in the same g_bus_remote_entries table the fanout walks.

Connection-loss supervision (GH #233 steps 3–4). Publish fanout marks a locus-served connect entry lost on send failure (skipping it thereafter — down-window publishes drop, never falsely succeed) and pushes it onto a mutex’d pending list (lotus_bus_transport_mark_lost). The top of lotus_bus_queue_drain — owner thread, the only place failure handlers may run — drains the list (lotus_bus_drain_lost_transports): each handle goes to the codegen-registered dispatcher (lotus_bus_set_loss_handler, emitted only when main declares the matching on_failure) or straight to the structural exit (lotus_bus_transport_lost_fallback). The dispatcher invokes main.on_failure(main_self, transport_self, link_lost_violation) and, when the handler bumped __restart_count via restart (t), calls lotus_bus_transport_reconnect (re-runs connect-with-retry against the entry’s stored path; success clears lost). Codegen support: lotus.main.self global (stored at main-locus instantiation), lotus_bus_transport_bind_self (entry ↔ locus self, emitted after each connect instantiation).

Env-configured routes (LOTUS_BUS_CONFIG) have no source-level declaration to hang a locus on and keep the direct C path: lotus_bus_register_remote(subject, url, role) returns an i32 status (0 ok / -1 unrealizable — scheme/addr validation, socket/bind/listen, connect-retry timeout, thread spawn), with listener-side work done synchronously at registration (udp: parse + bind + multicast join via lotus_bus_udp_listener_setup). On failure the entry is popped (no dead slot for fanout to silently skip) and lotus_bus_load_config routes into lotus_bus_binding_fail. Config-route unix listeners share lotus_bus_unix_serve, so they re-arm identically. Normative contract: spec/semantics.md, “The publish contract”.

Adapter dispatch. At codegen, an adapter binding instantiates the adapter locus into the program-lifetime payload arena (same m90 routing the -> LocusRef(L) return path uses), resolves the locus’s send method’s fn pointer, and registers the (self, send_fn) pair with the runtime via lotus_bus_register_remote_adapter. The runtime stores both in lotus_bus_remote_entry_t’s adapter slot. Outbound fanout packages the wire bytes as an Hale-level Bytes value (built via lotus_bytes_from_buf against the lazy global payload arena) and indirect-calls send_fn(self, subject, bytes). No vtable lookup is needed at the runtime layer — codegen resolved the method at binding-emit time.

Adapter inbound (m105). Adapters receiving wire-bytes from their protocol layer call std::bus::__local_dispatch( subject, bytes); the primitive backs onto lotus_bus_dispatch_wire, which looks up the subject’s registered deserialize fn in g_bus_entries (same table the publish-side fanout consults), reconstructs the struct-layout bytes, and fans into local subscribers via lotus_bus_local_dispatch. Symmetric to the unix reader- thread path; out-of-band recv loops (any code holding wire bytes for a bound subject) can use this too.

SPSC observation ring (GH #244). A single-producer fixed-slot ring over CALLER-PROVIDED memory, exposed as lotus primitives (lotus_spsc_init / _emit / _note_drop / _set_tag_b / _read) and the raw all-Int Hale surface std::ring::__spsc_*. Built for observation planes (the iris observer attaches to these rings inside an shm segment, read-only, from a foreign process), so the layout is a STABLE documented contract:

Consumer cursors and overrun counters live OUTSIDE the shared segment (caller-owned); external readers never write the ring. This ring is the convergence target for iris’s observation protocol (its PROTOCOL.md pre-freeze sketch is this layout with tag_a/tag_b as sched_id/current_locus).

SHM ring substrate (Form K5). POSIX shared- memory ring backing the zero-copy bus route. Six C primitives in runtime/lotus_shm_ring.c, linked unconditionally so user programs that bind a topic to a zero_copy route resolve cleanly:

Layout in SHM: a 64-byte cache-aligned header (lotus_shm_ring_header_t — magic, slot_size, slot_count, atomic seqno) followed by N slots of slot_size bytes each. Header magic + sizes are validated on attach to catch ABI mismatches across binaries pinned to the same ring name.

Foreign-layout consumer (Proposal B). Two more primitives read an externally-defined ring described by a ring_layout (see semantics.md § “Foreign rings”), rather than the native LRSRNG1 shape:

The producer side (Proposal B M3a) mirrors these:

Field reads/writes are host-native endianness (the foreign producer and Hale are both little-endian x86-64). v1 is byte_records only; the slots framing kind and a zero-copy writable producer view are post-v1.

v1 scope: single-producer, multi-consumer; in-memory delivery is in scope (POSIX shm_open works intra-machine cross-process). Multi-producer (CAS-based claim), back-pressure / timeout modes, and named-ring registry are post-v1. The Hale-side fallible(ClaimError) signature is reserved for those; v1’s claim() never actually fails.

Lifecycle / cleanup. Both lotus_bus_register_shm_ring (publisher) and lotus_bus_register_subscriber_shm_ring (subscriber) register a single atexit hook on first call. The hook:

  1. Signals every subscriber reader thread to stop via an atomic should_stop flag.
  2. pthread_joins each reader thread, ensuring no in-flight handler is interrupted.
  3. Frees the subscriber state allocated by the registration call.
  4. lotus_shm_ring_closes every ring opened in this process — which shm_unlinks the ones this process created (owns_unlink=1), keeping /dev/shm/ clean across restarts.

The atexit hook runs on a clean process exit (return from main, exit(3)). Signal-driven termination (SIGTERM, SIGKILL, _exit) bypasses atexit and leaves the SHM namespace entry behind until reboot or manual shm_unlink. A future v1.x SIGINT/SIGTERM handler can fold into this same teardown.

Constraint: subscriber handlers must not call exit(). The handler runs on the reader thread; calling exit() from inside the handler invokes atexit on the reader thread, which then attempts to pthread_join itself (undefined behavior). Use _exit() if a handler needs to terminate the process immediately.

Codegen surface for K5 lands in K4 (route-selection + slot-locus synthesis) and K6 (subscriber view + epoch guard). K5 ships the substrate; user code can’t reach these symbols directly without going through the slot-locus surface a zero_copy binding produces.

Closure-test infrastructure

Section titled “Closure-test infrastructure”

Perspective infrastructure

Section titled “Perspective infrastructure”

Failure handling

Section titled “Failure handling”

Native observation emission (iris P4, 2026-07-27)

Section titled “Native observation emission (iris P4, 2026-07-27)”

With LOTUS_OBS=1 the runtime publishes an iris-protocol observation segment (/hale-obs-<pid> + registration file per PROTOCOL v0.2 — the contract lives in the iris-observer repo; layouts mirrored from its reference implementation. 0.2, 2026-08-12 downstream handoff: the header gains model_hash at 0x80 — the topology artifact’s shape_hash, stamped by the CLI from the same bundle it typechecks, so a consumer can establish the RUNNING binary was built from the model it joins against; 0 = unstamped harness build. And the per-binding backpressure cells reserved since v0 are now written: queue_depth (cell 3, a last-write-wins gauge of the kernel send-queue occupancy sampled at send time), send_block_ns (cell 4, accumulated transport-send duration), and retries (cell 5, reconnects) — counters-tier, so a consumer falling behind shows as depth climbing and block time accruing BEFORE anything drops) and the runtime’s own choke points emit records: BUS_PUBLISH / BUS_DELIVER at every dispatch flavor (dynamic, static-devirt, cross-thread wire; deliver is enqueue-time at v0), NET_SEND / NET_DELIVER at the transport fanout/reader with per-binding monotonic seqs (what iris seq-matches into cross-process edges), LOCUS_BIRTH / LOCUS_DISSOLVE (both cooperative and pinned lifecycle paths; dissolve rides the emit_locus_arena_destroy chokepoint) and RESTART at reconnect. Cost contract: env unset = one predictable branch per probe (g_obs_state); enabled-but-unobserved (observer_count == 0) = counters only, no ring writes; ring emission is SPSC via the #247 primitive with one ring per emitting thread (TLS assignment; overflow threads count into ring_drops_total). On the observer-count 0→1 rising edge the live-locus table replays as LOCUS_BIRTH records so a late-attaching observer reconstructs the tree. Knobs: LOTUS_OBS_RINGS (default 8), LOTUS_OBS_SLOTS (default 4096), LOTUS_OBS_WIRE (default off — see NET seq semantics: LOTUS_OBS alone never alters the wire; the cross-process edge header rides the wire only when the operator opts the whole fleet in with LOTUS_OBS_WIRE=1). v0 notes: manifest ids are registration-order; publisher locus attribution on BUS_PUBLISH is unattributed (0); pinned births render parent=root. lotus_obs.c is its own TU; the arena TU’s probes are weak-guarded so helper binaries compiling lotus_arena.c alone still link.

Field-hardening (iris handoff 2, 2026-07-27, v0.11.13): NET_SEND now fires on the UDP multicast fanout path (it continues before the stream-transport probe, so multicast publishers emitted only deliver-side records — the seq matcher had nothing to pair); LOCUS_BIRTH is emitted BEFORE a locus’s field-default init and carries real parentage (the parent registers before its children look it up — post-birth placement rendered every tree flat), and pinned children register on the spawning thread so a park before their first probe doesn’t hide them; BUS_PUBLISH stamps the publishing locus from a codegen-set TLS (the C dispatch doesn’t know self); topic shape_hash is fnv(subject, canonical payload structure) — field names + coarse type tags in declared order, never the declaring type’s local name — so two binaries sharing a subject fuse into one manifest row; and each ring re-emits EPOCH every 1024 records so a high-rate ring that wraps its anchor never reconstructs timestamps from a stale base (the ~2^64 ns readings).

NET seq semantics (iris handoff 3, v0.11.14): NET_SEND and NET_DELIVER carry w1 = origin:16 | seq:48 where origin is the SENDER’s per-process identity and seq is the SENDER’s per-binding counter — the receiver echoes both verbatim from the wire, so a send and its delivers pair on (origin, seq) across segments (the cross-process edge). Before this, the receiver stamped its LOCAL delivery counter, which sums across senders on a multicast subject — the send seq never equalled the deliver seq and iris rendered zero edges. (Stream transports are unicast — one sender per connection — so origin 0 + the framed wire seq already pairs correctly.)

Edge-emission correctness (iris handoff 4, v0.11.15):

Flavor completeness + quiet-process replay (iris handoff 5, v0.11.18):

BUS record w1 layout (iris handoff 7, v0.11.21): BUS_PUBLISH / BUS_DELIVER pack w1 = locus:20 (bits 44..63) | seq:44 (low) — PROTOCOL §8, executable reference iris emitter/protocol.h (obs_bus_w1). The emitter had these fields transposed (locus low, seq shifted) since handoff-3; attribution was computed correctly and packed unreadably, and the contract tests stayed green because they decoded with the emitter’s own layout. The tests now vendor protocol.h’s decode, so emitter and consumers cannot disagree silently.

Adapter ingest completeness (iris handoff 8, v0.11.22): the Hale-owned-wire ingest (std::bus::__local_dispatchlotus_bus_dispatch_wire_inbound) now carries the full probe trio the C reader threads have. The wrapper peels the self-describing obs wire header when the producer emitted one (magic-guarded — a headerless or non-Hale producer is byte-for-byte unaffected, and headered bytes reaching a Hale adapter previously failed deserialization outright), emits NET_DELIVER echoing the wire (origin, seq) (or (0, local per-subject seq) when headerless — countable, not pairable), and the plain dispatch_wire fanout gains the per-target BUS_DELIVER its keyed sibling received in v0.11.18. Adapter bindings register lazily in the manifest (aux = 2). Dynamically-ingested planes now form producer→consumer edges and attribute subscriber loci exactly like statically configured listens.

Attribution ordering + the adapter inbound path (iris handoff 6, v0.11.20):

Lossless recording mode (GH #296 Phase 1)

Section titled “Lossless recording mode (GH #296 Phase 1)”

LOTUS_OBS_RECORD=<path> opts a run into recording: the observation plane stops being a sampler and becomes a flight recorder. Implies LOTUS_OBS=1; the recording counts as an attached observer, so ring emission is on from the first probe with nothing mmapping the segment. Opt-in on the same terms as everything else here — unset, the lowering and the wire are byte-for-byte the unobserved build (the flag resolves in the same constructor as lotus_obs_live and gates behind it everywhere).

Three deltas from observer mode, all in the service of one rule — a recording never drops (a dropped record is a replay that diverges silently):

Durable recording / crash-prefix recovery (Phase 5). Named precisely: this is a durable flight recorder, NOT a write-ahead log — the application is never gated on the recording reaching stable storage, so a crash can lose records the application already executed past. (A load-bearing WAL grade — persist, fence, then permit delivery — is a possible later mode; nothing here claims it.) What IS guaranteed: the drain appends whole frames in stream order and flushes every sweep, so the on-disk file is an exact prefix of the stream at all times, and the header identity (model_hash, exec_digest, policy flags) is stamped eagerly by the drain as soon as the ctor-sequenced setters have run, not only at finalize. A run that dies without teardown therefore leaves an artifact that is attributable and exact up to one torn frame at the tail. Default recording rides the page cache (survives a process crash, not power loss). LOTUS_OBS_RECORD_DURABLE=1 is the power-loss grade: every flushed sweep passes fdatasync, the parent directory is synced at file creation (a fresh NAME is not durable until its directory entry is), the finalize trailer itself is fdatasync’d before close (a clean exit followed by power loss must not demote a finalized recording to a truncated one), and the grade is recorded in the header’s policy flags (bit 1).

Consume records (private recorder namespace). BUS_DELIVER is enqueue-time by design and lands on the publisher’s ring — correct for fanout accounting, structurally unable to say in what order a consumer actually ran its handlers. Under recording, every dequeue-driven handler invoke (main-queue drain, coop-pool drain, pinned mailbox drain, async coro start) stamps a consume record on the consuming thread right before the handler runs, carrying the delivery identity: the target locus in the record’s id field and the full 64-bit msg_id in w1. Its ring position is the per-consumer delivery order — the thing a replay serves back. Pairing rule: per queue, the k-th consume is the k-th queued delivery. The synchronous direct-dispatch flavors deliberately emit no consume record: their handler runs at the BUS_DELIVER position, which is the consumption point.

Recorder events (CONSUMER/CONSUME/ENQ/JOURNAL) live on process-private per-thread rings in their own event namespace, never in the public observation segment: iris protocol ekinds 8/9/11/12 mean SUPERV_TRANS/PLACEMENT/BINDING_UP/BINDING_DOWN, and an observer attaching to a recording process must never decode recorder bookkeeping as lifecycle transitions. In the recording file, private-ring entries carry the high bit in their ring field, so the two namespaces can never be confused.

Delivery identity (Phase 2). Every queued delivery carries a deterministic pub_id = consumer_id:16 | per-publisher-thread seq:48 (full width — the review-round guards fail loudly rather than wrap), re-derivable by a re-executed run without global coordination. Consumer ids are stable across runs, unlike pthread ids: main = 1, cooperative pool workers = 16 + registration index, pinned locus threads = 64 + obs instance id; threads with no stable identity (ingress readers) get run-unique anonymous ids, never a shared fallback. Payload bytes are captured once per queued publish under a stable subject hash (manifest topic ids are registration-order and racing publishers register in either order). Flags: bit 0 = external wire ingress; bit 1 = raw in-process struct bytes — an ABI snapshot (String/Bytes fields are pointers, padding is uninitialized) that consumers must compare by size only; canonical per-topic recording codecs are the staged fix. Wire captures are canonical bytes, unflagged. The synchronous direct-dispatch flavors deliberately capture nothing: a closed-world same-thread call cannot carry external input, and re-execution re-derives its payloads.

Input journal (Phase 3). Under recording, every user-facing nondeterministic read is journaled per consumer as ONE unified stream carrying the read’s kind and its exact encoded arguments (length-framed; replay memcmps them — hashes proved collision-prone), so a changed env name, arg index, rand bound, or read length is the first named divergence, never a silently substituted value. Env values are withheld by default: names, existence, and lengths are recorded; the value itself requires LOTUS_OBS_RECORD_ENV=full, the artifact header says which policy applied, and a withheld read replays as a named withheld divergence. The interposed set: std::time::now, std::time::monotonic[_ns] (now a named primitive, lotus_time_monotonic_ns — it used to be inline clock_gettime IR that nothing could interpose), std::rand::next_int (per call — the global RNG’s mutex makes seed-replay unsound across threads), std::os::getrandom, and the std::env surface. Internal runtime clocks and config getenvs are deliberately not journaled.

File format v0.3 (PRE-STABLE). 96-byte header (magic HALEREC0, version, pid, ring geometry, epoch anchors, model_hash, a 32-byte framed-SHA-256 exec_digest, and policy flags), tagged entries — tag 0: 24-byte ring record; tags 1/2/3: payload / journal / meta blobs (32-byte header + bytes padded to 8; meta carries the run-stable topic and public-ring identity maps) — and a 16-byte trailer (HALEEND0 + entry count). A recording is clean only when the ENTIRE artifact validates: exact parse to the trailer position and a matching entry count, enforced by the CLI reader and independently by the runtime loader (one open + fstat + private mmap, checked arithmetic, per-kind value shapes) — trailer magic at EOF alone proves nothing. Identity fields are stamped eagerly by the drain and re-stamped at finalize (a prelude binding registration can probe — creating the header — before the stamps run; the two stamps agree). A trailer-less artifact is a crash-truncated recording: refused by default, but admissible as a PREFIX with hale replay --allow-truncated (LOTUS_REPLAY_ALLOW_TRUNCATED=1) — the loaders stop at the first incomplete frame, report the parsed extent and dropped tail bytes, and replay that prefix; execution past the tape’s end falls back live and is counted, per the degrade-never-refuse rule. Under --diff a truncated baseline compares as a prefix — and the runtime verdict agrees with the comparator: recorded-history-exhausted events (journal reads past the tape, deliveries past the consume stream, consumers the recording never saw) count into a separate post_prefix_live_fallback status key, never into the divergence totals; mismatches BEFORE exhaustion stay divergences either way. A trailer-FINALIZED artifact keeps the exact full-parse + count checks: with a finalize present, any truncation is corruption.

Replay (hale replay <recording> <program.hl>). Admission, strongest check first: the recording’s exec_digest — a framed SHA-256 over the toolchain source hash (compiler + runtime + stdlib implementation, via the stale-CLI build hash), the CLI version, build options, and every source file’s full path, length, and contents — must match the recompiled program exactly; shape_hash is structural compatibility only, the secondary check. An unstamped recording is refused without --allow-unverified-model. Stated residue: the digest cannot see the LLVM/libc toolchain outside the hale binary; a post-link binary digest is the staged stronger form. Safe by default: replay re-executes real side effects, so the typed effect rows gate admission and fail CLOSED on syscall, ffi, unclassified (“may do anything”) — refused without an explicit --allow-live-effects (coarse by class granularity — over-refusal is the safe direction; per-primitive replay classes are the staged refinement). bindings blocks are no longer in that list: under replay the wire is hermetic (below), so a bound topic cannot reach the live world. LOTUS_REPLAY=<path> then serves journaled reads back per consumer in recorded order; a read past (or mismatching) the recorded history falls back live and is counted — replay degrades, never refuses (RFC Q6) — with a divergence summary at exit and a machine-readable verdict (LOTUS_REPLAY_STATUS) the CLI consumes: journal misses, order holds, unconsumed journal entries, and unconsumed deliveries all count, and any of them fails --diff. Each consumer’s queued deliveries are re-consumed in the RECORDED order (Phase 4): dequeued cells that arrive ahead of their recorded turn are held per-consumer and released in order, with a bounded hold (1s) after which the oldest held cell is released and the miss counted, so a genuinely divergent replay reports rather than deadlocks.

Async pools replay (Phase 6). The nondeterminism of a where async_io pool is its drain’s SCHEDULING: which cell starts when, which parked coro resumes when (epoll readiness order), and when a timed park expires relative to both. Under recording, every such decision is stamped on the pool worker’s private ring as a step (ASYNC_START/ASYNC_RESUME/ASYNC_EXPIRE; coros are named by birth ordinal — the index of the START that created them, stable across runs because start order is enforced). Under replay the drain is DRIVEN by that stream instead of the clock: START steps reuse the phase-4 cell gate (start order is consume order); RESUME steps wait for the named coro’s readiness, parking early-ready coros aside until their turn; EXPIRE steps resume the named coro with the timed-out sentinel immediately — the recording already proves the deadline fired at this point in the sequence, so replayed sleeps fast-forward rather than re-waiting. A step unsatisfiable within the hold bound counts an async-schedule divergence and is skipped (degrade, never deadlock) — and a skipped START retires both its birth ordinal and its consume slot (review round 2: ordinals belong to the recorded START slots, not to whichever cell happened to start next — without retirement one missing delivery shifts every later coroutine into an earlier recorded identity, and the pinned consume expectation makes every later START mismatch; with it, later steps that name a retired slot skip immediately and the divergence stays local).

Coverage, named (review round). A dry tape hands the pool back to the live drain — never silently. The artifact carries a capability bit (header flag 4: this runtime records async schedules); the states are distinct and reviewable: a pre-phase-6 artifact gets a one-shot coverage note (“recording predates async-schedule support”); a truncated tape’s remaining schedule is the stated coverage boundary (excluded from the divergence totals); a FINALIZED capable artifact whose tape runs dry while the re-execution keeps doing async work counts each such action as async_post_tape, and schedule steps left unconsumed at exit count as unconsumed_async_steps — both machine-readable status keys and divergence rows. --diff compares the per-consumer async step streams (kind, value) bidirectionally, exactly like the consume and journal streams — “byte-identical output” alone is weaker than “the schedule matched”. Boundary, stated: the SCHEDULE replays; the DATA of unjournaled I/O does not — a coro resumed at its recorded turn re-executes its recv against the live world (syscall-class, gated), and bindings ingress arrives via the Phase-5 injector.

Hermetic wire + ingress injection (Phase 5). Hermeticity is a binding-kind capability, not a blanket assumption: the native unix:// and udp:// transports (and the transport-locus form) are suppressed and injectable; a backend with no replay class fails closedshm_ring in particular is refused by the CLI (backend named) and independently at the runtime’s shm-open seam, so a replayed or fed process never creates, opens, or mutates live shared memory. Adapter loci are user logic and re-execute as such (governed by their inferred effects).

For the covered backends, under replay a binding is suppressed at realization: the entry exists (a husk, transport NULL — the same shape the reclaim path already leaves), but no socket is created, nothing binds, nothing connects, and outbound fanout to bound subjects sends nothing (the publish is still recorded and compared under --diff).

In the listeners’ stead, injection runs as an explicit boot phase: codegen emits one lotus_replay_start_ingress() call at the main locus’s boot/run boundary — params children born, bindings realized or suppressed, every boot subscription registered, run() not yet entered — where the runtime snapshots the subscription registry ON the main thread (injector workers never read the live registry) and spawns one worker per recorded ingress source (the pub_id’s consumer bits), each carrying its source’s recorded consumer identity so a --diff verify recording aligns per-consumer streams with the original instead of collapsing every listener onto one injector identity. Fresh anonymous claims are floored above the recorded range. The snapshot IS the coverage boundary: a subscriber the program only creates during run() (spawned or accepted) is not visible to injection — such tape entries classify as late_subscription_uncovered (the injector join rescans the live registry at teardown to distinguish them from genuinely absent subscribers), a stated boundary rather than a silent unmatched. Injection is keyed by full identity: each tape record carries its complete subject string and the subject’s canonical payload shape (FNV-32 alone is collision-prone and is kept only for reporting); a record whose shape does not match the live program’s shape for that subject is refused as incompatible, never fed as a plausible wrong value. Only messages the original application accepted enter the tape — the wire capture runs after the deserializer says yes, so identity allocation agrees between record and replay and rejected traffic is never re-fed.

Pacing. Strict replay injects one message, then waits (bounded by the phase-4 hold) until that message’s recorded consumes have all been re-consumed — unpaced injection can shed at bounded queues (bounded(N, drop_old/drop_new), on_full policies) before the dequeue-side order gate ever sees the cell, which no amount of holding can undo. Feed mode is deliberately unpaced.

Accounting. Every tape entry ends in exactly one class: injected, rejected-by-deserializer, unmatched-subject, incompatible-shape, or unprocessed-at-shutdown (the injector is joined at teardown and its remainder classified); injector start failure is its own class. Under strict replay every non-injected class is a divergence, in the machine-readable status (ingress_*, injector_start_failure keys) and the exit summary.

Feed mode (hale replay --feed, LOTUS_REPLAY_FEED=<path>) — same recorded ingress, changed code, live nondeterministic environment. Stated that precisely because “same inputs” would overclaim: feed reproduces recorded binding ingress only — time, randomness, env, and argv stay live. The recording is consumed as an input tape: recorded ingress injected, wire hermetic, and nothing else of replay applies — no journal serving, no order enforcement, no model admission (a model-hash mismatch is reported informationally; feeding a tape to changed code is the point), no --diff/--at (there is no recorded schedule to compare against). The effects gate still applies: --feed bypasses identity admission, never effect safety — a program whose frontier reaches syscall/ffi/unclassified still requires --allow-live-effects alongside --feed. Mutually exclusive with LOTUS_REPLAY. Injected deliveries carry the new run’s own identity. The exit report classifies every tape entry, and an unfed remainder (unmatched, incompatible, unprocessed, start failure) fails the run by default--allow-unmatched-feed is the explicit acceptance of a partial feed.

--diff (strict replay only — feed rejects it, above). --diff records the replay (under the original’s env policy) and compares bidirectionally: per-consumer queued consume streams (target locus + msg_id), per-consumer PUBLIC bus streams aligned by subject and stable consumer id via the meta maps — which is what makes synchronous direct dispatch visible — payloads both ways (topic, flags, bytes; raw metadata by declared size), and per-consumer journal streams (kind, exact args, withheld state, value; grouped per consumer because the drain interleaves concurrent threads’ entries in incidental order). Any runtime divergence fails --diff through the verdict. --at <n> stops (SIGSTOP) at the nth consume process-wide (meaningful for one consumer); --at <consumer-id>:<ordinal> is the stable multi-consumer form. Replay implies observation (identity rides the obs machinery).

Two honest limits, stated rather than implied: the recorded interleaving is reproduced per consumer, not globally — cross- consumer wall-clock alignment is not a replay property; and the injected tape covers bindings ingress — raw user-level socket reads are syscall-class live effects (gated), and adapter loci re-execute their own protocol logic. Fleet-scale replay (multiple binaries against one composed tape, with #262’s plan admission for replay-under-a-different-plan) is the next milestone.

Time

Section titled “Time”

I/O — minimal

Section titled “I/O — minimal”

Text + string primitives (v1.x adds)

Section titled “Text + string primitives (v1.x adds)”

Process control

Section titled “Process control”

stdout buffering

Section titled “stdout buffering”

stdout is line-buffered for the lifetime of the program, regardless of whether it’s attached to a TTY or a pipe. The main prelude calls setvbuf(stdout, NULL, _IOLBF, 0) once before any user code runs.

The default libc behavior (fully-buffered when stdout isn’t a TTY) silently dropped output for any program that printed then blocked on a syscall — println("READY"); accept_loop(); made “READY\n” invisible to a piped consumer until the buffer filled or the program exited. Test oracles, supervisors waiting for a READY handshake, and log tailers all hung. Line-buffering matches Python python -u discipline and Go’s default; \n- terminated println calls flush immediately under any stdout target.

stderr is line-buffered by POSIX already; the runtime doesn’t touch it.

What’s NOT in the runtime (lives in stdlib instead)

Section titled “What’s NOT in the runtime (lives in stdlib instead)”

These are bundled with the toolchain (no separate install) but require explicit import std::....

Form-vec runtime (v1.x-FORM-1)

Section titled “Form-vec runtime (v1.x-FORM-1)”

The @form(vec) form lowers to a contiguous growable buffer implemented in C. See spec/forms.md for the form contract and synthesized method set; this section documents the runtime shape.

C struct layout

Section titled “C struct layout”

Each @form(vec) locus’s heap slot lowers to an inline struct:

typedef struct {
size_t cap; // allocated capacity (elements)
size_t len; // number of valid elements
char *buf; // contiguous element array
} lotus_vec_<T>_t;

The <T> suffix is conceptual — codegen monomorphizes per cell type T, but the runtime primitives operate on the common prefix layout via void * casts. All lotus_vec_*_t typedefs share the {cap, len, buf} prefix.

Primitive functions

Section titled “Primitive functions”

Defined in crates/hale-codegen/runtime/lotus_arena.c (v1.x-FORM-1 PR4):

Function Behavior
void lotus_vec_init(void *v) Zero-init: cap=0, len=0, buf=NULL
void lotus_vec_push(void *v, size_t es, const void *x) Append; doubles cap on overflow
int lotus_vec_get(void *v, size_t es, int64_t i, void *out) Bounds-checked read; returns 1=OK, 0=out-of-bounds
int lotus_vec_set(void *v, size_t es, int64_t i, const void *x) Bounds-checked in-place write; returns 1=OK, 0=out-of-bounds (does not extend the vec)
int lotus_vec_pop(void *v, size_t es, void *out) Returns 1=OK, 0=empty
int64_t lotus_vec_len(void *v) Element count
int lotus_vec_is_empty(void *v) 1=empty, 0=non-empty
void lotus_vec_destroy(void *v) free(buf); called at locus dissolve
void lotus_vec_sort_int(void *v) In-place ascending sort of an int64_t-cell vec via qsort
void lotus_vec_sort_float(void *v) In-place ascending sort of a double-cell vec; NaN treated as equal-to-anything
void lotus_vec_sort_string(void *v) In-place ascending sort of a char *-cell vec under strcmp ordering
void lotus_vec_sort_by(void *v, size_t es, int (*cmp)(const void *, const void *, void *), void *cookie) qsort_r wrapper; cmp is a codegen-synthesized per-(cell_type, direction) trampoline

es (elem_size) is the cell type’s size in bytes — codegen passes sizeof(T) at each call site.

Growth policy

Section titled “Growth policy”

Failure shapes

Section titled “Failure shapes”

Form-hashmap runtime (v1.x-FORM-4)

Section titled “Form-hashmap runtime (v1.x-FORM-4)”

The @form(hashmap) form lowers to an intrusive open-addressing hash table implemented in C. See spec/forms.md for the form contract and synthesized method set; this section documents the runtime shape.

C struct layout

Section titled “C struct layout”

Each @form(hashmap) locus’s pool slot lowers to an inline struct:

typedef struct {
size_t cap; // power-of-two slot count
size_t len; // live entry count
size_t key_size; // sizeof(K), set at init
size_t value_size; // sizeof(S), set at init
int key_type_tag; // 0 = Int, 1 = String
char *slots; // cap * (1 + key_size + value_size) bytes
} lotus_hashmap_t;

Each slot is 1 + key_size + value_size bytes:

[occupied: u8] [key: key_size bytes] [value: value_size bytes]

occupied = 0 means empty. Backward-shift deletion (no tombstones) — probes terminate at the first empty slot.

The C ABI is type-erased: codegen passes key_size / value_size at init time, and per-call sites pass raw void * key/value pointers. Codegen GEPs the indexed-by field on the caller’s side to derive the key pointer before each set.

Primitive functions

Section titled “Primitive functions”

Defined in crates/hale-codegen/runtime/lotus_arena.c (v1.x-FORM-4 PR4):

Function Behavior
void lotus_hashmap_init(void *m, size_t key_size, size_t value_size, int key_type_tag) Allocate cap=8 slots, zero them; freeze key/value sizes and key-type tag
void lotus_hashmap_set(void *m, const void *key, const void *value) Insert or replace; grow at load factor 0.7
int lotus_hashmap_get(void *m, const void *key, void *out_value) Bounds-checked read; returns 1=OK, 0=missing_key
int lotus_hashmap_has(void *m, const void *key) 1=present, 0=missing
int lotus_hashmap_remove(void *m, const void *key) 1=removed, 0=missing
int64_t lotus_hashmap_len(void *m) Live entry count
int lotus_hashmap_is_empty(void *m) 1=empty, 0=non-empty
void lotus_hashmap_destroy(void *m) free(slots); called at locus dissolve

Key types and hashing

Section titled “Key types and hashing”
key_type_tag Type Hash function Equality
0 (LOTUS_HASHMAP_KEY_INT) int64_t Knuth multiplicative (k * 0x9E3779B97F4A7C15) == on i64
1 (LOTUS_HASHMAP_KEY_STRING) const char * (NUL-terminated) FNV-1a over the bytes strcmp == 0, with pointer-identity fast path

Other key types (Bytes, custom structs, enum tags) are not supported at v1; codegen rejects @form(hashmap) with a focused diagnostic when the indexed-by field’s resolved type doesn’t map to one of these two tags.

Growth policy

Section titled “Growth policy”

Deletion policy

Section titled “Deletion policy”

Backward-shift deletion (no tombstones). After clearing the target slot, the runtime walks forward through the cluster and shifts any entry whose natural position is “before” the freed slot in the probe sequence. The cluster boundary is the first empty slot encountered. This keeps probe chains tight and lets find_slot terminate correctly without a separate tombstone marker.

Failure shapes

Section titled “Failure shapes”

Form-ring-buffer runtime (v1.x-FORM-5)

Section titled “Form-ring-buffer runtime (v1.x-FORM-5)”

@form(ring_buffer, cap = N) lowers a pool capacity slot to an inline lotus_ring_buffer_t and synthesizes a fixed-capacity FIFO surface (push / pop / len / is_full). The cap is baked in at lotus_ring_buffer_init from the form annotation arg; the backing buffer is malloc’d once at locus birth and never grows.

C struct layout

Section titled “C struct layout”
typedef struct {
size_t cap; // fixed at init; never changes
size_t head; // index of oldest element (next pop)
size_t len; // current element count, 0..=cap
size_t elem_size; // bytes per element
char *buf; // cap * elem_size bytes
} lotus_ring_buffer_t;

Codegen emits the matching LLVM inline struct on the locus’s pool slot; the slot’s struct field IS the ring buffer (no indirection). Element-size is sizeof(T) from the cell type’s LLVM size_of.

Primitive functions

Section titled “Primitive functions”

Defined in crates/hale-codegen/runtime/lotus_arena.c (v1.x-FORM-5):

Function Behavior
void lotus_ring_buffer_init(void *rb, size_t cap, size_t elem_size) malloc(cap * elem_size); head=len=0
int lotus_ring_buffer_push(void *rb, const void *src) 1=pushed, 0=full; wraps modulo cap
int lotus_ring_buffer_pop(void *rb, void *out) 1=popped, 0=empty; advances head
int64_t lotus_ring_buffer_len(void *rb) Current element count
int lotus_ring_buffer_is_full(void *rb) 1=full (len==cap), 0=not
void lotus_ring_buffer_destroy(void *rb) free(buf) at locus dissolve

Failure shapes

Section titled “Failure shapes”

Native codegen defaults

Section titled “Native codegen defaults”

What the compiler emits for a native hale build:

Diagnostic + tuning env vars

Section titled “Diagnostic + tuning env vars”

A small set of env vars toggle runtime instrumentation and glibc tuning hooks. All are opt-in; unset (the default) keeps the runtime quiet.

Env var Effect
LOTUS_ARENA_LOG_BIG_CHUNKS=<N> Logs every arena chunk + libc allocator (malloc / realloc / calloc / mmap) >= <N> bytes to stderr with size, monotonic seqno, and 8-frame backtrace. Use 1 (= 1 MiB) as a shortcut; any positive decimal byte count works (e.g. 4096). Each event labeled by source: arena_big_chunk, malloc_big, realloc_big, calloc_big, mmap_big. Note: only fires on the fresh-malloc path; chunks recycled from the per-thread pool bypass this hook — use LOTUS_ARENA_LOG_CHUNK_ATTACH for the full picture.
LOTUS_ARENA_LOG_CHUNK_ATTACH=<N> Logs every chunk attachment to ANY arena — both fresh-malloc (chunk_attach_malloc) AND per-thread-pool-recycled (chunk_attach_pool) paths — when cap >= N. Use 1 for “log every chunk attachment”. Each event additionally prints arena=<ptr> kind=<root|sub> label=<resolved>: root means the chunk attached to a top-level locus-lifetime arena (a leak class if it grows); sub means a subregion (method scratch / free-fn body) that will recycle to the pool on destroy. label walks the subregion→root chain and looks up the root in the residency registry — requires LOTUS_ARENA_RESIDENCY=1 to populate the label map. Filter kind=root label=<name> to isolate the actual arena-growers. Shares the LOTUS_ARENA_LOG_BIG_MAX_EVENTS cap with the big-chunks logger.
LOTUS_ARENA_LOG_BIG_MAX_EVENTS=<N> Caps the log at <N> events per process. Default 200. Set to 0 for unlimited (useful when watching low-rate sub-MiB allocation patterns over a long window).
LOTUS_CHUNK_POOL_STATS=1 Dumps per-thread chunk-pool hit / miss / store / overflow counters to stderr at process exit. Diagnostic for “pool isn’t recycling” symptoms — pairs hits vs misses, stores vs overflows. The atexit handler runs on the main thread; counters are __thread so the dump is that thread’s view.
LOTUS_GLIBC_ARENA_MAX=<N> Calls mallopt(M_ARENA_MAX, <N>) at startup. Caps glibc’s per-thread malloc arena count. 1 forces a single arena (max contention, min virtual-address fragmentation); higher <N> trades contention for parallelism. Useful belt-and-suspenders against the per-thread arena heap-segment proliferation glibc default tuning can produce on long-running daemons. Unset keeps glibc’s default.
LOTUS_BUS_PAYLOAD_ARENA_CAP=<N> Overrides the lazy-global bus payload arena’s byte cap (default 64 MiB). When the cap fires, lotus_arena_alloc returns NULL and the existing alloc-fail paths (empty_global / alloc_failed violation) surface degraded service rather than OOM-killing the process.
LOTUS_ARENA_RESIDENCY=1 Registers every top-level arena (locus __arenas, g_bus_payload_arena, the program-wide global) into a side-table at creation time with a 24-frame construction backtrace. std::process::dump_arena_residency() walks the live set and emits one line per arena to stderr — bytes / chunks / parent / label, sorted by bytes desc — with the construction backtrace. Subregions (method scratch) are skipped; they destroy at method exit and don’t accumulate residency. Atexit also dumps, but post-dissolve fires after all loci tear down — useful only for the global arena’s final state. Long-running daemons should call dump_arena_residency from a heartbeat / checkpoint tick so locus arenas are sampled while still alive.
LOTUS_CHUNK_POOL_PREFILL=<N> Per-thread chunk-pool pre-fill on first touch. Default 32 (= 2 MiB resident per scheduler thread). Set 0 to disable. Bumps the pool’s steady-state floor so brief bursts don’t drain to zero and miss into malloc; the trade-off is per-thread resident memory.
LOTUS_TSAN=1 Read at build time (by the codegen’s build_executable, not at runtime). When set, the emitted clang command passes -fsanitize=thread for both the C runtime compile and the binary link, and skips the -Wl,--wrap=malloc/realloc/calloc/mmap shim surface (TSAN intercepts malloc itself; the wrap’d LOTUS_ARENA_LOG_BIG_CHUNKS diagnostic is silently no-op under TSAN). The resulting binary runs ~5-15× slower; use only for race-hunting workloads. The C runtime embeds an empty __tsan_default_suppressions hook at link time so no external suppression file is needed; all originally-flagged substrate races (bus queue drain, arena destroy, coop pool worker, env-var lazy-init) have been fixed and the suppression list is empty. Opt-in tests live behind #[ignore] and the env var (see crates/hale-codegen/tests/form_hashmap_lockfree_tsan.rs).
LOTUS_LTO=1 Read at build time. Opt-in full-LTO native build: the Hale module is emitted as bitcode and the lotus runtime TUs compile with -flto, so the clang -flto -O3 -fuse-ld=lld link inlines the runtime hot paths (arena / string / shm_ring) across the TU boundary into the Hale callers. A few percent on allocation/coordination-heavy code, neutral on vectorized loops (host tuning preserved via per-function target-features). Off by default — ~3–4× slower link, requires lld. Native non-sanitizer only; --wrap shims survive (lld resolves them before LTO codegen). See Native codegen defaults above.
LOTUS_BUS_LOG_UNMATCHED=1 Surfaces silent no-key-match drops in lotus_bus_local_dispatch_keyed (Phase 3 routing keys). When set, each publish that matches no where key == ... subscriber for the topic emits a single stderr line citing subject, key, and the per-topic subscriber counts (specific vs unkeyed). Off by default — the silent-drop is correct for on_unmatched: swallow topics in steady state, but during bring-up the lack of any signal is load-bearing on debug cycles. Implied by LOTUS_BUS_LOG_DROP=1.
LOTUS_BUS_LOG_DESERIALIZE_DROP=1 Surfaces silent drops in the udp:// reader thread when (a) no deserializer is registered for the inbound subject, or (b) the deserializer returns <= 0 (size mismatch, bounded-read failure). Emits one stderr line per drop naming the subject, the payload size, and (when applicable) the deserializer’s return value. Off by default; the silent-skip on cross-routed multicast noise is the correct steady-state behavior. Same env-gated pattern as LOTUS_BUS_LOG_UNMATCHED for keyed-dispatch misses. Implied by LOTUS_BUS_LOG_DROP=1.
LOTUS_BUS_QUEUE_CAP=<N> Caps the cooperative bus dispatch queue, each per-pinned-locus mailbox, and each cooperative pool’s queue at N cells (default 8192; floor 64; rounded up to a power of two; read once). v0.9.0 footprint change: the pinned mailbox and cooperative-pool queues are now lock-free MPSC rings (Vyukov bounded ring + signal-only-when-parked wake), and a fixed-size lock-free ring pre-allocates its cap up front rather than growing to it — so each pinned subscriber mailbox and each cooperative pool now costs ~4.3 MB resident at the default cap (vs the prior grow-as-needed). With the typical handful of pinned loci / pools this is a few-to-low-tens of MB; lower LOTUS_BUS_QUEUE_CAP for pinned-/pool-heavy programs to shrink it (the rings honor it identically). When a producer hits the cap it back-pressures instead of growing without bound (GH #125) — every message is still delivered. The mechanism: a single-threaded producer on the cooperative queue inline-drains it to free space; a cross-thread producer to a full ring blocks (a fenced producers-waiting handshake) until the single consumer drains a slot; a handler self-publishing to its own full ring spills to a consumer-thread-local overflow list (it can’t block on itself). The cross-cooperative-pool shared queue path (multiple drainers, no single consumer) is the remaining non-lock-free path — a follow-on. Lower the cap to tighten the bound / footprint; raise it to reduce drain bursts at the cost of resident memory.
LOTUS_UNIX_STREAM=1 GH #231: forces the unix bus transport into framed SOCK_STREAM mode on Linux (the Darwin default — macOS has no AF_UNIX SOCK_SEQPACKET). Wire format per message: [u64 LE payload len][u64 LE seq][payload]; the seq is per-connection monotonic from 1, reset per accepted peer, and the receiver counts gaps (seq_gaps counter — GH #236’s loss-computability primitive). Set for EVERY process on a socket: framed and SEQPACKET ends don’t interoperate (a mismatch trips the 8 MB length sanity cap with a diagnostic naming the likely cause). Primary use: Linux CI/test coverage of the macOS code path.
LOTUS_BUS_COUNTERS_DUMP=1 GH #236 (observability groundwork): prints one stderr line per remote binding at teardown — [bus counters] subject=... kind=... role=... sent= delivered= bytes_sent= bytes_delivered= send_failures= dropped_lost= waits= rearms= reconnects=. The counters are plain relaxed atomics bumped at the transport choke points (fanout send, serve-loop dispatch, re-arm, reconnect) and exist as the substrate for the iris observer; the dump is the operator/test surface until an in-process consumer ships. dropped_lost counts publishes made while a connect binding was in the lost/reconnecting window (GH #233 — drops the publish contract makes deliberate and visible). waits counts publishes that parked in or wait through that window instead (GH #255 phase 1). buffered_early / dropped_early (GH #468) count boot-window wire messages buffered before any matching subscriber registration existed, and the subset dropped (buffer eviction at the 64-msg/1 MiB per-binding cap, a deserialize failure at flush, or a subject that never gained a subscriber by teardown). Socket-buffer occupancy is intentionally not a counter: it’s a poll-time SIOCOUTQ query against the live fd.
LOTUS_BUS_LOG_DROP=1 Broad superset for diagnosing “publish appears to succeed but handler doesn’t fire” symptoms. Implies LOTUS_BUS_LOG_UNMATCHED + LOTUS_BUS_LOG_DESERIALIZE_DROP AND covers additional silent-drop sites the narrower vars miss: lotus_bus_dispatch’s serialize-fn-returns-<=0 case, the local-fanout (lotus_bus_dispatch_wire + lotus_bus_local_dispatch) zero-matching-subscribers case, per-entry deserialize-returns-<=0 on the local-fanout path, and the no-post-target case (mailbox / coop_pool / global queue all NULL on a matched entry). Each line names the call site, subject, and relevant size / index info so a bus-heavy repro can identify exactly which silent-skip is firing. Reach for this first when investigating bus-drop friction; the narrower vars stay supported for their specific bring-up scenarios.
LOTUS_BUS_QUIESCE_MS GH #468: bound on the main-exit ingress quiesce (default 500, 0 disables it). At every main-exit point, before pools join and loci dissolve, LISTEN binding fds half-close and their readers drain kernel-accepted data to true EOF through the still-intact registry; this bounds how long exit waits for a wedged reader. Exceeding the bound is loud (stderr names it) and drops whatever stayed undrained. Test-only siblings LOTUS_BUS_TEST_BOOT_HOLD_MS / LOTUS_BUS_TEST_READER_STALL_MS deterministically stretch the boot-registration window / the reader’s descheduled window (the loss canaries in binding_ingest_468.rs use them); never set them in production.

Every top-level arena is created via lotus_arena_create_labeled(name) and carries an immutable human-readable label string. The codegen passes the locus name (e.g. WsClient, __lib_metrics_metrics_MetricMap); the program-wide global is labeled lotus.arena.global; g_bus_payload_arena labels itself. The label is the load-bearing identifier in the residency dump; backtraces resolve via -rdynamic for cases where the label alone isn’t enough.

The arena-chunk pool and -wrap=malloc family ship in every binary unconditionally; the env vars are zero-cost when unset (one int read + one branch per allocation). The -rdynamic link flag is similarly unconditional so backtrace symbols resolve without addr2line.

Runtime size budget

Section titled “Runtime size budget”

The runtime should be small enough that a hello-world program binary is < 1 MB statically linked, and < 100 KB if dynamic linking against libc. This is a target, not a guarantee.

The framework’s discipline enables this: no GC, no metadata overhead per allocation, region-based MM compiles to bump allocators. Comparable to C in size, with ergonomics closer to Erlang.

Open questions for runtime

Section titled “Open questions for runtime”