Runtime
Reference material, synced from the compiler repo’s
spec/. The guide is the gentler path in.
What every compiled Hale binary always ships with. Always-
loaded; not optional; no import needed; the substrate every
Hale program depends on.
This document distinguishes the runtime (always there) from
the standard library (stdlib.md, importable but bundled).
Go’s distinction between runtime and other stdlib packages is
the model: runtime is automatic; stdlib is explicit.
Naming note: The language is Hale; the runtime/ substrate concept is called lotus, and the C-runtime symbols stay
lotus_*(per project memory). When this doc says “lotus” it means the substrate; “Hale” means the language proper.
What’s in the runtime
Section titled “What’s in the runtime”Memory
Section titled “Memory”-
Region allocator. Per-locus arenas, hierarchical, freed on dissolution. Bump allocation within a region; no per-object metadata; no GC. The framework’s lotus structure provides the scope; the allocator just respects it.
-
Per-method scratch. A locus method body (lifecycle / user-fn / mode) opens a per-call subregion of
self.__arenaat entry and destroys it at every return — unless the body provably allocates nothing and returns a by-value scalar (or Unit), in which case the scratch is elided (2026-06-28; an optimization with no observable effect — there’s nothing to reclaim, so skipping the subregion just removes amalloc/freeper call). Transient allocations made inside the body —to_string,Stringconcat,std::str::*/std::json::*/std::bytes::*results, format-string composition — route through the scratch viacurrent_arena_ptr()and get reclaimed at method exit. Heap-typedself.X = exprstores deep-copy intoself.__arenabefore the store so persisted state outlives the scratch destroy. Heap return values are deep-copied into the caller’s arena via a fn-local snapshot oflotus_caller_arena_or_global()taken at the method’s entry block. Callers publish theircurrent_arena_ptr()vialotus_set_caller_arenaimmediately before each method call (same TLS contract as stdlib primitives). Without this, long-runningrun()loops accumulated every transient allocation into the locus’s lifetime arena — a real workload measured multiple MB/sec of growth on a hot message-dispatch path, OOM within minutes under a typical container cap. Seespec/memory.md“Phase-4 per-method scratch reclaim” for the full design (invariants, cost model, interaction with the cross-seed-segv routing). -
Per-projection-class allocation strategy. Rich → simple arena; chunked → arena with per-coordinatee sub-regions; recognition → recpool, sub-mode-typed at the declaration site (see “Recognition pool allocators” below). Selected at compile time per locus.
-
Free-list within parent for bookkeeping reclamation. When a coordinatee dissolves, its bookkeeping slot in the parent’s arena is reclaimed via a per-arena free-list (chunked-class loci) or periodic defrag (high-churn loci). Reclamation is per-arena, bounded, deterministic — never stop- the-world. Coordinatee sub-regions remain pristine arenas freed wholesale on dissolution.
-
F.22 capacity-slot allocators. Each
pool X of T;/heap Y of T;declaration on a locus adds a per-instance allocator. The C runtime ships two symbol families:Family Surface Backing lotus_pool_*create(cell_size, cell_align) -> pool*,acquire(pool) -> cell*,release(pool, cell),destroy(pool)Linked list of chunks; each chunk is one malloc holding N contiguous cells. Free-list threads through the cells themselves (each free cell stores the next-free pointer at its base). Chunks grow geometrically (initial sized so one chunk fits in a host page when stride permits, else 16 cells; doubling; capped at 4096). Cell stride = max(cell_size, sizeof(void*)) aligned to cell_align. v1.x-17: initial chunk cell count is max(16, page_size / cell_stride)capped at 4096 — sysconf(_SC_PAGESIZE) queried once and cached; falls back to 4 KiB on systems where sysconf returns implausible values.lotus_heap_*create(cell_size, cell_align) -> heap*,alloc(heap) -> cell*,free(heap, cell),destroy(heap)Doubly-linked live list with intrusive header (prev/next pointers) sitting just before each cell. free()unlinks in O(1);destroy()walks the list and frees every still-live cell wholesale.Both allocator families are type-erased at the C ABI (sizes
- aligns are i64 args). Cell alignment is 8 bytes uniformly
in v0; loci with cells requiring >8-byte alignment (e.g.
AVX-aligned types) are not supported. Per F.22 §“Slot
lifetime”, slot init runs after slot 0 / arena and destroy
runs in reverse before slot 0 / arena. See
spec/semantics.md“Capacity slot lifecycle and dispatch (F.22)” for the language-level surface andspec/memory.md“Capacity slots (F.22)” for the lotus-substrate framing.
- aligns are i64 args). Cell alignment is 8 bytes uniformly
in v0; loci with cells requiring >8-byte alignment (e.g.
AVX-aligned types) are not supported. Per F.22 §“Slot
lifetime”, slot init runs after slot 0 / arena and destroy
runs in reverse before slot 0 / arena. See
-
Recognition pool allocators (v1.x-3). A locus with
: projection recognition(cap=N, <sub_mode>)allocates one recpool at instantiation; child loci accepted by that parent draw their arena from the pool instead oflotus_arena_create _subregion. Two symbol families ship in v1:Family Surface Backing lotus_recpool_fixed_*create(cap, cell_bytes) -> recpool*,acquire(recpool) -> arena*,release(recpool, arena),destroy(recpool)One contiguous block of cap × cell_stridebytes. Each cell carries an INLINElotus_arena_t+lotus_arena_chunk_theader at its front followed bycell_bytesof payload — the cell IS the child’s arena. Bitmap (uint64_t[ceil(cap/64)]) tracks occupancy; acquire scans the lowest unset bit via__builtin_ctzll. Release clears the bit so the slot is reusable. The returned arena hasfixed_size=1, solotus_arena_allocreturns NULL on overflow (caller routes to the closure-violation channel).lotus_recpool_slab_*create(cap, slab_bytes) -> recpool*,acquire(recpool) -> arena*,release(recpool, arena),destroy(recpool)One lotus_arena_twith an initial chunk ofslab_bytesandfixed_size=1so it never grows. Everyacquirereturns the SAME arena pointer — children share the bump space and per-child release is a no-op. The whole slab frees at parent dissolve vialotus_arena_destroy(slab_arena).capis recorded but not enforced at the C layer (codegen’s birth-cap check bounds concurrent children; the slab is a memory budget).Both families return
lotus_arena_t*fromacquireso child body code stays projection-class-agnostic per the F.22 architectural invariant — the samearena_allocpath handles fresh, subregion, fixed-cell, and shared-slab children. The codegen dispatch at child dissolve picks the matchingreleasefn via the synthetic__recpool_release_kinddiscriminator (0 = regulararena_destroy, 1 = fixed_cell release, 2 = shared_slab release). v1 shipsfixed_cellandshared_slab;spilloverandsummary_onlyparse + AST through but reject at typecheck with av1.x pendingdiagnostic (the spillover malloc-fallback machinery and thesummary_only“no child arena allocation” type-system rule are separate work).
Lifecycle
Section titled “Lifecycle”- Lifecycle dispatcher. Invokes
birth → run → drain → dissolveper locus; invokesaccepton coordinatee attachment; invokeson_failureon child failure with the parent’s policy. - Interest-based ownership (accept bubbling).
accept(c: I)collects not only a direct child but the nearest such acceptor for anI{}instantiated anywhere in the subtree: when a locus instantiatesI{}and its direct enclosing locus does notaccept(I), ownership bubbles to the nearest enclosing ancestor that does (innermost-wins). Resolution is entirely static — there is no polymorphic locus instantiation, so the closed-world instantiation graph fixes every owner edge at compile time; no runtime ancestor walk. Backward-compatible by construction: innermost-wins selects the direct parent whenever it accepts, so no existing parent↔child edge changes; bubbling only adds an owner where a child would otherwise be a transient throwaway. AnI{}with no accepting ancestor stays transient — ownership is opt-in viaaccept, and the absence of an owner is never an error. Same-tower bubbling costs nothing beyond the direct-parent case (the owner pointer is a constant for a singleton owner, or threaded down the birth chain for multiple owner instances — giving each owner instance its own isolated collection — then the ordinary accept path). A cross-pool owner (e.g. amain locusregistry collecting entities spawned on a worker pool) is served by an async handoff over the bus queue: the child is born on the owner’s thread and reclaimed by the owner’s same-thread cascade, so a cross-poolI{}is fire-and-forget — it may only be a bare statement; using the instance as a value is rejected at compile time. - State machine enforcement. A locus can’t accept after drain has begun, can’t run before birth completed, etc. The runtime tracks state; transitions are rejected if they violate ordering.
drain()cascades depth-first. Callingdrain()on a locus first recursively drains all its children (depth-first), waits for them, then drains itself. SIGINT triggersdrain()on the runtime root, cascading through the whole process tree. No separate cascade syntax —drain()is always cascading. For locus-typed param fields specifically (F.29), the codegen walksLocusReffields in declaration order at the cascade-teardown sites (ephemeral scope-exit and deferred-flush) and calls each child’s drain BEFORE the outer locus’s own drain. The subsequent dissolve cascade runs the outer’sclosures → dissolvebody next, then per childclosures → dissolve → arena_destroy, then outer’s arena_destroy. Pinned-thread tail still skips the cascade per the v1 trade-off. Anaccept’d child is reclaimed on its OWN run-completion /terminatewhen it is a flow (see “Per-child reclamation” below) rather than waiting for the parent’s cascade.- Per-child reclamation. An
accept’d child’srun()is posted to its pool as a coro, run through a synthesized__coop_pool_run_<L>wrapper. When that run() completes, the wrapper reclaims the child — drain → (for a flow) the parent’srelease(owner, self)→ dissolve → arena/recpool release — iff the child is a flow (some declared locus hasrelease(c: ThisType)) OR it set the__drain_requestedlatch viaterminate;. A non-flow (“resident”) child whose run() merely returns is NOT reclaimed (it lives to parent dissolve). The reclaim runs on the child’s own pool worker while its arena is valid;emit_locus_arena_ destroyis idempotent (NULLs__arena), so a later parent-dissolve of the same locus no-ops. At pool shutdown a coro may still be PARKED (a listener inaccept()); the wakeable park handles it: a per-pool wakeeventfdin the pool’s epoll letsshutdown_allunblock a worker sitting inepoll_wait(-1)(the condvar broadcast can’t), so the worker returns from the drain and the join completes instead of hanging. The parked coros are then abandoned — their stacks freed without resuming them — because a forever-looprun()(while true { accept }) cannot be cooperatively unwound without the loop checkingself.draining(a future refinement), and the process is exiting anyway. Per-child reclamation proper (terminate / flow run-completion) never needs this: there the coro returns fromrun()on its own. - Classic-pool blocking-accept shutdown. A
classic (non-
async_io) pool worker blocked in a blockingaccept(2)inside a locus’srun()(e.g.std::http::Serverorstd::io::tcp::Listenerplaced on a plaincooperative(pool = X)) can’t be woken by the wakeeventfd(there is no epoll on a classic pool) — so two rules keep its teardown clean: (a) the classicacceptpolls the listen fd with a short timeout and checks its pool’s shutdown flag, so it returns a sentinel-1onceshutdown_allis signalled and the stdlib accept loops (Server/Listener) break out of their forever loop; and (b) the main locus joins all pool workers before dissolving itsparamsfields, so a worker still executing a pool-placed field’srun()can never touch that field’s arena after it’s freed (the alternative — freeing first — is a use-after-free; the alternative join-without-(a) is a hang). Together these let a program whosemainrun() returns while a classic-pool server child is live shut down cleanly rather than hanging or segfaulting. - Teardown delivery contract (GH #253, 2026-07-27). A
dissolving parent joins its own pinned children — mailbox
shutdown,
pthread_join, then a bus drain — BEFORE cascading its field children’s drain/dissolve, and the fn-exit flush likewise joins subscription-less pinned entries before any cooperative entry’s teardown. Consequence: events a pinned worker publishes in its final moments (up torun()return) are delivered to sibling subscribers, in any declaration order — a parent whoserun()returns immediately no longer silently drops its workers’ last publishes (the hale-bun install-fanout shape). This is the pinned mirror of the pool-worker join rule above, and it also closes the lifetime hazard of destroying the owner’s arena (which holds the pinned children’s self structs) while their threads still ran. What still drops, by contract: a publish made after its last subscriber has dissolved — e.g. from a pool-placed publisher whose queued work outlives the process teardown, or from adissolve()body publishing to a subscriber that tore down earlier in the same cascade. Those cells are discarded via the deregister-on-dissolve invariant (never dispatched to freed memory);LOTUS_BUS_LOG_DROP=1surfaces them. A workload that must not lose such events coordinates completion explicitly (“exit when done” — poll a tally / await a completion event) rather than relying on teardown ordering. Deferred-path completion (Crumb batch-4, 2026-07-28): the contract holds regardless of which teardown path the parent takes. Previously a bus subscription on the main locus made it long-lived → deferred, and the deferred flush processed the parent (cascading its subscriber fields) BEFORE joining its own pinned children — the exact inversion the contract forbids, triggered by the one-line addition of asubscribeon the root, dropping every in-flight result silently. A deferred parent’s own pinned entries are now re-ordered after its own frame entry, so the reverse-order flush joins + drains them while every subscriber field is alive — identical semantics to the eager path. - Recovery primitives.
restart,restart_in_place,quarantine,reorganize,bubble,dissolve,drain— all language keywords; runtime implements the actual effects.
Scheduler — multi-scheduler cooperative
Section titled “Scheduler — multi-scheduler cooperative”Lotus uses a multi-scheduler cooperative model (closest existing analog: Erlang BEAM, not Go’s M:N). The reasons are framework-discipline:
- Lateral-access prohibition is physical, not just typed. Within a single cooperative scheduler, sibling loci cannot run concurrently — only one locus is executing at a time per scheduler. There is no thread of execution that could attempt a lateral memory reference. The compile-time type rule (“vertical-only flow”) is reinforced by the substrate.
- Substrate-cell atomicity is naturally aligned. Cooperative yield points — between message-handler invocations, between lifecycle phases, on bus dispatch — are exactly where the substrate-cell boundary lives. No preemption inside a substrate-cell because the runtime can’t preempt at all; it only switches at yield points.
- Per-scheduler region allocators. Each scheduler is single-threaded, so its allocator state is naturally per-scheduler with no synchronization. Lock-free by construction.
- Failure-traversal is a call-stack walk on one scheduler. No cross-thread synchronization for parent-catches-child failure when both are on the same scheduler.
Concurrency comes from running multiple cooperative schedulers in parallel (one per CPU core, by default). Loci belong to a specific scheduler; cross-scheduler communication uses the bus just like cross-process communication. Loci may be migrated between schedulers transparently for load balancing because all their communication is bus-mediated already.
Specifically:
- One scheduler per CPU core at startup, configurable.
- Cooperative yield points: between handler invocations,
between lifecycle transitions, on bus message dispatch, on
explicit
yield(rare, for long-running computations). Plain fn exit is NOT a yield point — and since 2026-07-02 a proven-non-allocating fn’s exit provably skips the queue drain (a non-allocating body cannot have published; payload copies allocate). A cooperative compute-only loop that leaned on helper-call returns for delivery never had that guarantee and must use an explicityield;. - No preemption within a scheduler. A locus’s handler runs to completion or an explicit yield.
- Cross-scheduler is bus. No shared memory; no locks.
- Failure-traversal: if parent and child are on the same
scheduler, failure-traversal is a stack walk. If different
schedulers, the failure is delivered as a typed bus message
to the parent’s scheduler, which dispatches to
on_failure.
Placement classes (per-locus execution strategy)
Section titled “Placement classes (per-locus execution strategy)”Just as projection class governs a locus’s memory strategy,
placement class governs its execution strategy. Placement
is a deployment seam, not an intrinsic property of the locus
(see spec/decisions.md § F.31). Placement entries
live in a placement { } block on main locus only, parallel
to bindings { } for bus topology:
main locus App { params { gateway_a: Gateway = Gateway { venue: "venue-a" }; gateway_b: Gateway = Gateway { venue: "venue-b" }; metrics: MetricsServer = MetricsServer { port: 9100 }; ui: Renderer = Renderer { }; } placement { gateway_a: pinned(core = 1); gateway_b: pinned(core = 2); metrics: cooperative(pool = io); ui: cooperative(pool = render); // unspecified main-locus params → cooperative(pool = main) }}Placement remains honestly bimodal: either a locus shares a cooperative pool (an OS thread running a cooperative drain loop) or it owns its own OS thread. There is no third position.
| Class | Yield discipline | Resource |
|---|---|---|
cooperative(pool = X) (default for unspecified main-locus params, with X = main) |
Yields between substrate cells (handler exit, lifecycle transition, bus dispatch, time::sleep, explicit yield). time::sleep slices into ≤100ms intervals and folds in lotus_bus_queue_drain for the locus’s pool after each slice, so cells posted by other threads deliver mid-loop even during a long keep-alive sleep. Handler bodies are atomic. |
Shares pool X’s OS thread with other cooperative loci placed on the same pool. |
pinned / pinned(core = N) / pinned(cores = A..B | A..=B | {a, b, c}) / pinned(node = N) / pinned(l3 = name) / pinned(..., replicas = K) |
No yield to siblings; owns its OS thread. Bus events to/from cross-thread boundaries via formal mailbox post. | Dedicated OS thread. core = N pins it to one CPU; cores = ... (Phase 1a) sets its mask to a core set; node = N / l3 = name (Phase 1b) set the mask to a NUMA node / cache domain from topology { } (and bind the arena there); replicas = K (Phase 1c) fans into K single-threaded instances, one per core. The OS schedules freely within the mask. Linux-only; best-effort no-op elsewhere. |
Pool inference rule. The cooperative pool set is inferred
from cooperative(pool = X) references in the placement { }
block. The runtime spawns one OS worker thread per inferred
pool name beyond main (which is always the program’s main
thread). No separate threads { } declaration block at v1 —
when per-pool attributes (priority, affinity, realtime hint)
become useful, the block lands then as a typed extension. Pool
main exists in every program regardless of whether
placement { } references it.
Nested-instantiation inheritance. Placement entries apply
only to top-level main locus params fields. Loci
instantiated nested in another locus’s body (in birth /
run / lifecycle methods, or as let-bound children) inherit
the parent’s pool. There is no way to spell “this nested child
runs on a different pool than its parent” — that would require
the nested-instantiation expression to carry placement, which
would re-mix the deployment and intrinsic layers F.31 separates.
Topology block (Phase 1b). A main locus may
also declare a topology { } block — a declare-only
description of the host’s core partition, a sibling deployment
seam to placement { } / bindings { }:
topology { reserve cores 0..2; // held back for the OS / main node 0 { l3 fast { cores 4..8; } // a CCD / shared-L3 group l3 slow { cores 8..12; } } node 1 { l3 heavy { cores 12..16; } }}A placement { } entry then targets a domain: pinned(node = 0) sets the thread’s affinity mask to node 0’s core set (the
union of its L3 domains — here {4..12}), and pinned(l3 = fast) sets it to the named domain’s cores ({4..8}). The
compiler resolves the domain to a concrete core set at compile
time (closed-world: node ids and domain cores are literals),
reusing the same cpuset affinity mechanism as pinned(cores = ...). Validation (unique node ids, globally-unique L3 names,
non-overlapping domains, no domain/reserved overlap, and every
pinned(node/l3) referencing a declared domain) is static. L3
domain names go through the identifier rule, so a hard keyword
(e.g. bulk) can’t name a domain.
Replicas (Phase 1c). pinned(..., replicas = K) is the
parallelism sugar: it fans the field into K single-threaded
instances, replica i pinned to one core of the affinity set
(round-robin — pinned(cores = 4..12, replicas = 8) puts replica
i on core 4 + i; more replicas than cores wraps; with no
affinity the K instances are OS-scheduled). This is deliberately
not a multi-worker pool — a cooperative pool is one consumer
thread, and the lock-free rings, bus devirtualization, and
single-threaded-method guarantee all rest on that invariant.
Parallelism instead comes from more single-threaded units, each
its own single consumer, so every invariant survives. replicas
is pinned-only (K cooperative loci on one pool would share a
thread, which isn’t parallel) and composes with the topology
targets (pinned(node = 0, replicas = 4) fans across node 0’s
cores with each replica’s arena bound to node 0). Codegen emits K
instantiations at the field’s init site; all K register their bus
subscriptions (a subscribed topic fans out to every replica) and
all K are joined + dissolved at parent teardown via the deferred-
dissolve frame. The replicas are non-addressable — there is no
field[i] surface; they are workers that pull from the bus or run
their own loop.
Thread + memory co-location. A pinned(node = N) /
pinned(l3 = name) locus binds not just its thread but its
memory: its arena is created via
lotus_arena_create_labeled_on_node, which flags the arena’s
NUMA node, and every chunk that arena grows is mmap’d
(page-aligned) and bound to the node with the mbind syscall
(MPOL_BIND) before first touch — so pages fault in on the
node regardless of which thread touches them first (the locus
struct is instantiated on main but runs on its own pinned
thread). Sub-regions inherit the node, so a node-pinned locus’s
method scratch — the dominant per-invocation allocation —
lands on its node too. mbind is invoked as a raw syscall, so
this adds no libnuma dependency; the whole feature is
zero-cost for programs that don’t opt in (an unbound arena, the
default, takes the ordinary malloc / chunk-pool path
byte-for-byte). Best-effort and Linux-only, exactly like
pinned(core = N): an mbind the box can’t honor (node absent,
capability denied) falls back to first-touch, and on non-Linux
hosts the arena allocates normally. (Huge-page-backed chunks
and node binding don’t currently combine — a node-bound arena
uses regular pages; a follow-up.)
Single-threaded-method invariant. A locus’s methods may be invoked only on the OS thread that owns its placement’s pool. Cross-pool method calls and lateral field accesses go through the bus’s existing copy-and-condvar dispatch machinery, which already crosses thread boundaries safely. The typechecker walks the static call graph from each top-level placement entry, propagates pool ownership through method receivers, and rejects calls that cross pools without going through the bus. This is the substrate enforcement that makes M:N safe — without it, multi-pool deployments would silently race on locus arenas (which are unsynchronized bump allocators).
The single shared bus payload arena is the deliberate
exception to “one owning thread per arena”: it is reachable
concurrently from any pool, because some stdlib primitives
allocate their result there directly rather than into a
per-locus arena (e.g. std::io::tls::recv_bytes, which always
targets it regardless of the caller’s per-thread scratch). That
arena therefore carries an internal lock on its bump — two
pinned loci calling such a primitive at once do not corrupt each
other’s allocations. Per-locus arenas keep the lock-free bump;
only this one shared arena pays for the lock.
Why no “greedy” class
Section titled “Why no “greedy” class”A natural temptation is to want a third option: “shares a pool’s thread but doesn’t yield.” That would be a bimodality violation. Cooperative already guarantees handler-level atomicity — no preemption within a substrate cell — so the only thing such a class could add over cooperative is “don’t yield between cells either.” But that means leaving the shared pool entirely. The place you go when you leave is your own thread. That’s pinned.
Latency-critical work, or anything that genuinely shouldn’t
share with neighbors on its pool, is signaling that it belongs
on its own pool (placed cooperative(pool = some_quiet_pool))
or on its own thread (placed pinned). The first option is
new in F.31: pools partition the cooperative substrate so
“shouldn’t share with siblings” no longer forces pinned — but
the underlying bimodality (cooperative-pool vs pinned-thread)
holds.
time::sleep drain semantics
Section titled “time::sleep drain semantics”The codegen lowering of std::time::sleep(d) slices the request
into intervals of at most 100ms and ends each slice’s
EINTR-retry loop with an inline call to lotus_bus_queue_drain
against the program-wide cooperative queue (plus a pinned-mailbox
drain). The total wall-clock sleep is preserved (the slices sum to
d); sleeps of 100ms or less take exactly one slice, so the common
case is unchanged. A cooperative subscriber looping
run() { while !self.bail { std::time::sleep(100ms); // ... loop body sees handlers fired during the sleep }}receives cells posted by other threads — unix-bound reader
threads, pinned publishers via lotus_bus_local_dispatch, etc.
— right when each sleep returns, without an explicit yield;.
The drain is idempotent, so existing code with sleep; yield;
stays correct.
Which path delivers to whom. The queue drained here is the
program-wide in-process cooperative queue, drained on the main
thread — so this sleep-loop drain delivers in-process cells to a
cooperative subscriber on the main pool. It is not the only
delivery path: cross-process topics (udp:// / unix:// bound via
LOTUS_BUS_CONFIG) are delivered by the transport reader thread,
which dispatches directly into the subscribed handler set and
reaches a cooperative locus on any pool — provided that pool’s
thread is free to run the dispatch. So a non-main cooperative
subscriber receives reliably as long as it doesn’t monopolize its
pool thread with a blocking call. (Pinned loci are a third path: a
per-locus mailbox drained at each sleep/yield.) The failure mode
is a cooperative subscriber whose run() blocks — the dispatch
can’t run and its handlers never fire; that blocking-and-subscribing
combination is what the dead-bus-receiver rule rejects (see
“Type-check rules” in spec/semantics.md), not non-main
placement on its own.
Long sleeps no longer starve main-pool handlers. Before the slicing, the drain happened only after the whole sleep returned, so the natural keep-alive idiom
run() { while true { std::time::sleep(60s); } } // on the main threadstarved every pool = main bus handler for 60s at a time: a
main-pool subscriber registers with coop_pool == NULL, so the
wire/reader-thread dispatch path lands its cells on the global
cooperative queue (g_bus_queue_for_remote, the same object only
main drains). A 60s blocking sleep on main left those cells
unserviced for the full 60s — indistinguishable from “the handler
never fires.” Slicing keeps the queue serviced ~10×/s during any
sleep, so a main-pool handler that republishes onto a topic an
async-pool subscriber listens to (e.g. a udp-reader-fed producer
forwarding to per-connection writers) now flows promptly regardless
of how long main is asleep.
The pinned-mailbox path is unchanged: pinned subscribers wake
on lotus_mailbox_post’s condvar broadcast regardless of what
the cooperative scheduler is doing.
Owner-executed handlers (2026-07-15, downstream handoff)
Section titled “Owner-executed handlers (2026-07-15, downstream handoff)”Two runtime rules restore the single-threaded-locus invariant (F.31) dynamically — the compiler already enforces it for direct calls, but two bus paths used to violate it under load (reproduced as a SIGSEGV in a 10k msg/s ingest bench):
- The global cooperative queue is drained only by its owner
thread (
main, recorded at queue creation). The scope-exit flush emitted at the end of every fn/method body — and the sleep-slice /yielddrains — calllotus_bus_queue_drainon whatever thread ran the body; on any thread but the owner the call is now a no-op. Previously a pinned publisher’s flush would execute a main-pool subscriber’s handler on the publisher’s thread, concurrently withmain’s own drains (the locked drain releases the queue mutex before each handler invocation) — two threads inside one locus. - Payload deserialization happens on the subscriber’s owner thread. The non-flat dispatch paths deserialize each published payload into the subscriber’s arena (Task-11 arena routing); for a target owned by a different thread this write used to happen on the publisher’s thread — an unlocked cross-thread write into a foreign arena. Now a cross-thread publish enqueues the wire bytes plus the deserialize fn in the cell, and the owner materializes (deserializes into its own arena) at drain, just before invoking the handler. Same-thread targets keep the deserialize-at-dispatch fast path, so single-pool programs are unchanged.
Consequences: a main-pool subscriber’s handlers run only on
main (at sleep-slice / yield / scope-exit drains — worst-case
~100ms after a cross-thread publish, per the slicing above);
pool subscribers run only on their pool’s worker; pinned
subscribers only on their own thread. Flat (pointer-free POD)
payloads are exempt from rule 2 — a verbatim byte copy writes no
arena. Delivery and FIFO order per subscriber are unchanged.
Item B from the 2026-05-21 friction log (a cooperative
publisher’s <- to a pinned subscriber) is resolved as of
2026-06-01 — the earlier “drains only at dissolve in some
configurations” was a sequencing effect, not a lost wakeup.
The mailbox condvar path is correct: a pinned subscriber whose
run() returns proceeds into the blocking
lotus_mailbox_drain_one, and a cooperative publisher’s <-
wakes it via the not_empty broadcast — confirmed by
coop_to_pinned_mid_program::pinned_returning_run_drains_mailbox.
The residual constraint is inherent to a single pinned thread:
a pinned subscriber with a long-running run() that never
returns or yields cannot drain its mailbox during run() —
the one thread is busy in the loop. Such a run() must reach a
cooperative yield point (time::sleep / yield) for the
TLS-cached lotus_mailbox_drain_pending to service the mailbox
(the coop_to_pinned_mid_program sleep-loop test), or let
run() return so the post-run blocking drain takes over. This
is a property of the bimodal model — “pinned owns its thread,
no yielding between cells” — not a bug; a pinned subscriber that
must receive bus traffic while busy should yield in its loop.
Cross-binary flows (unix / shm_ring with their own reader
threads) land in g_bus_queue and benefit from the sleep-folded
drain above.
Long-running cooperative children: placement closes Item D
Section titled “Long-running cooperative children: placement closes Item D”Pre-F.31, declaring a long-running cooperative child as a
params field (metrics_server: std::http::Server = ...)
serialized parent and child onto the main thread: the child’s
run() body never returned, so the parent’s run() never
started. The workaround was the sibling-in-main pattern —
hoisting the child to a top-level main locus param so it
ran “alongside” the parent rather than “inside” it.
Under F.31 the sibling-in-main pattern IS the canonical
shape, and it composes cleanly because placement { } lets
each sibling pick its own pool:
main locus App { params { gateway: Gateway = Gateway { }; metrics: std::http::Server = std::http::Server { port: 9100 }; } placement { gateway: pinned(core = 1); metrics: cooperative(pool = io); }}Parent’s run() no longer serializes against metrics’s
accept loop — they sit on different OS threads. To shut down
gracefully when one finishes (e.g. a duration-bounded gateway
exits), call metrics.shutdown() from the finishing locus’s
thread; the C-iii interruptible-accept work makes this the
supported pattern.
The nested-as-child shape (long-running cooperative locus as
a params field of a non-main locus) remains structurally
serialized — that’s a consequence of the nested-instantiation
inheritance rule (children share their parent’s pool by
construction). Nested long-running children are an antipattern
under F.31: hoist to main-locus siblings.
Typecheck enforcement. The compiler rejects
the antipattern at typecheck. A non-main locus with a non-trivial
run() body holding a params field of a locus type whose own
run() is also non-trivial — including std::http::Server and
the other entries on the known-long-running stdlib allowlist —
gets a hard error pointing at the canonical sibling-in-main +
placement fix. The runtime starvation that motivated this rule
is silent (the parent’s run() simply never executes), so the
type-side rejection is load-bearing: it converts a class of
hard-to-diagnose runtime bugs into a clear compile-time signal.
where async_io — green-I/O cooperative pools (F.35)
Section titled “where async_io — green-I/O cooperative pools (F.35)”The sibling-in-main fix puts each long-running child on its own
OS thread, which caps concurrent connections at one-per-pool. To
scale beyond that without spawning a thread per connection, a
placement entry may declare where async_io:
main locus App { params { listener: std::websocket::Server = std::websocket::Server { ... }; worker: WsWorker = WsWorker { ... }; } placement { listener: cooperative(pool = ws_accept) where async_io; worker: cooperative(pool = ws_workers) where async_io; }}where async_io opts the pool into green-I/O scheduling: the
pool’s worker drain loop integrates an epoll instance, and
blocking I/O syscalls inside locus methods on this pool park-
and-resume instead of blocking the OS thread. The user code
inside the locus is unchanged — recv_bytes(stream) reads the
same line of source whether the pool is async_io or not; the
substrate picks the right lowering at the syscall boundary.
One worker thread per named cooperative pool — a promise, not
an implementation detail (Crumb batch-3, 2026-07-28). Every
named cooperative pool (async_io or classic) has exactly one
OS worker thread for the lifetime of the program. Consequences a
consumer may depend on: every run(), bus handler, and parked-
coro resume for loci on the pool executes on that one thread
(coroutines interleave on it; they never migrate), so a
thread-affine C library — a JS engine, SQLite in serialized
mode, a GUI toolkit — placed on a named pool is entered from a
single thread by construction, and @export re-entry from
foreign code called on that thread stays on it. Concurrency
within a pool is cooperative only. If a scale-out mechanism ever
lands, it will be a new opt-in placement form; cooperative(pool = name) keeps the single-worker guarantee.
Because parking yields the shared worker, N reader loci that each
park on their own fd — the F.35 one-reader-per-signal shape —
are serviced concurrently by a single pool. Two invariants make
that multiplexing correct: (1) the drain loop starts a queued
run() the moment the running coro parks, not only when it
completes, so a long-lived reader never starves the readers
queued behind it; and (2) each coro’s caller-arena — the
thread-local that decides where its stdlib allocations (recv
result blobs, string builders) land — is snapshotted across the
park and restored on resume, so a coro that resumes after a
sibling ran (and perhaps dissolved) never allocates through an
arena the sibling has since torn down. Every blocking-recv
primitive on the pool honors invariant (1) by parking rather than
blocking recvfrom/read (the std::io::udp recv family joined
the tcp/tls siblings here in the 2026-07-15 downstream
handoff, and std::time::sleep joined them 2026-07-28 (Crumb
batch-5) via a timer-only park — a deadline with no fd, serviced
by the same expiry sweep — so sleeping coros overlap on one
worker and a sleep inside a handler never blocks the pool; the
timer half of an event loop now costs what the design always
claimed).
Each bus delivery to a subscriber on an async_io pool runs its
handler on a coroutine (a struct + a 64 KiB stack). Rather than
allocate and free that pair per delivery, the pool keeps a bounded
per-worker free-list (cap 64) of completed coro slots and reuses
them — a warm fan-out skips the per-dispatch stack allocation
entirely (2026-07-16). The free-list is worker-thread-local (no
lock) and drained at pool teardown, so a busy async pool retains up
to 64 × 64 KiB (~4 MiB) of coro stacks at steady state. Transparent
to user code — a pure allocation optimization, no behavior change.
Typecheck rules:
- All placement entries on the same named cooperative pool must
agree on
where async_io. The pool’s drain loop is one-or-the- other. where async_iois rejected onpinnedentries. Pinned loci own their own OS thread and have no shared drain loop to park on.where async_iois rejected on poolmain. The main pool runs inline on the binary’s primary thread, with no dedicated worker to integrate epoll into.
See spec/decisions.md § F.35 (forthcoming) for the
green-I/O substrate design + perf-axis trade-offs.
(Compare: rich / chunked / recognition projection classes are genuinely three-way because N≈10, N≈30, and N≈300 are different cost regimes at scale — memory has more genuine intermediate ground than time does. Placement, even with M:N pool partitioning, stays bimodal.)
Cross-class bus semantics
Section titled “Cross-class bus semantics”- Cooperative → cooperative, same pool: handler enqueues on the pool’s queue; runs at the next substrate cell on that pool’s drain loop. Sender never blocks.
- Cooperative → cooperative, different pools: cross-thread post via the destination pool’s queue; the destination pool’s drain thread wakes on the condvar broadcast (same machinery as the cooperative→pinned path). Sender never blocks.
- Any → pinned: cross-thread post via the pinned locus’s lock-protected mailbox. Sender never blocks.
- Pinned → any: cross-thread post; pinned publisher doesn’t block waiting for delivery acknowledgement.
Implementation status (m26 + m27 + m28a + m28b + m28c; F.31 pending)
Section titled “Implementation status (m26 + m27 + m28a + m28b + m28c; F.31 pending)”m25 wired the annotation through parse / typecheck / codegen.
m26 ships cooperative semantics; m27 ships pinned threads
(run-only); m28a lifts pinned to full lifecycle; m28b lights up
cross-thread bus mailboxes — pinned loci can subscribe and
publish, with cells routed across threads via per-locus
mailboxes; m28c adds optional CPU-core affinity via
pthread_setaffinity_np.
F.31 (2026-05-23, Phase 1-5 + Phase 4a shipped): the
placement-at-main surface and M:N cooperative pools. The
per-locus : schedule annotation is removed; the placement
choice moves to a placement { } block on main locus.
Cooperative subscribers can be partitioned across N pools
(N OS threads, each running its own
lotus_coop_pool_worker drain loop against its own per-pool
ring buffer); cross-pool bus dispatch reuses the m28b
condvar+memcpy machinery. A pool’s worker thread can take the
same affinity forms as pinned — cooperative(pool = X, core/cores/node/l3 = …), applied right after the worker spawns
(2026-08-12; entries naming one pool must agree, and the main
pool takes none). The single-threaded-method invariant
— a locus’s methods may be invoked only from its pool’s thread
or via the bus — ships as a typecheck rule (Phase 5).
Phase 4 v1 limit (handler-only cross-pool delivery). The
runtime ships pool-aware bus dispatch for now: a subscriber
whose enclosing locus is placed on a non-main cooperative
pool gets its handler invoked on that pool’s worker thread.
Lifecycle methods (birth / run / dissolve / accept)
still run on the main thread for cooperative-pool loci —
the codegen does NOT yet relocate them to the pool worker.
For state mutated only inside bus handlers this is enough to
honor the single-threaded-method invariant (the handler is
the only writer of locus state on the pool thread). State
touched by both lifecycle bodies and handlers on a non-main
pool is a Phase 4b concern; the typechecker doesn’t flag it
today because lifecycle methods are not user-callable in the
recv.method() shape that Phase 5 checks. Plan: Phase 4b
moves lifecycle dispatch onto the pool worker via the same
queue mechanism (post “run_init” / “drain_exit” cells at
instantiation / scope-exit boundaries).
Runtime pool inheritance for in-method-body instantiation
(2026-05-29). A locus instantiated inside a method or
bus-handler body that is itself executing on a pool worker
inherits that pool at runtime. Codegen has no static
placement name for such a locus (placement keys on main-locus
params fields only), so the run-posting site and the
subscription-registration site resolve the pool as: the
compile-time-known pool when the locus IS a placed main-locus
field, else the pool whose worker is currently on-CPU
(lotus_coop_pool_current(), which reads the per-thread
g_current_pool_tls; NULL on the main thread). When the
resolved pool is non-NULL the child’s run() is posted to it
(its own cell — and, on an async_io pool, its own parkable
coro, so a blocking recv in the child’s run() parks that
child’s coro rather than the spawning handler’s) and its bus
subscriptions are tagged with that pool so dispatch routes to
the right worker. This is gated on the child being owned
beyond the spawning scope — accept’d, an owned param
field, or returned. A handler-local let-bound long-lived
locus is not owned (its deferred dissolve fires at the
handler’s scope exit), so posting its run() would execute
after it’s dissolved; those keep the prior behavior
(synchronous run(), global-queue subscription). The
canonical N-dynamic-children shape (per-connection handlers,
per-tenant workers) is therefore an accept’d child whose
run() holds the recv loop — it multiplexes on an async_io
pool by construction. See spec/memory.md § “Owned
param-field child allocation” for the companion arena rule
that makes the owned child’s full subtree outlive the
spawning frame.
Adapter loci instantiated inline in bindings { Topic: AdapterLocus { ... }; } are NOT main-locus params fields
and so receive no placement { } entry. Their run() recv-
loops need a dedicated thread by construction; the substrate
places them pinned-equivalent implicitly (same m90 routing +
pthread spawn that pre-F.31 fired via the adapter’s
: schedule pinned annotation). The annotation goes away but
the behavior is preserved automatically — the bindings-inline
shape unambiguously signals “transport adapter with a recv
loop.”
The implementation notes below describe pre-F.31 shapes (m25
through m28c). They remain accurate for the cooperative-only-
on-main-pool case, which is the v1-compatible default when
no placement { } block is declared.
m26 (cooperative): Each <- enqueues (handler, self, payload_copy) cells onto a program-wide FIFO queue
(@lotus.bus_queue.global) instead of running handlers
inline. The scheduler drain loop pops cells one at a time
and invokes the handler — handler-atomic per substrate cell,
with cooperative yields BETWEEN cells rather than nested call
frames. Handlers may publish more events; drain continues
until empty.
Drain runs at the start of every flush_dissolve_frame —
before any long-lived locus dissolves — so subscribers process
pending cells while still alive. Plus an explicit yield;
statement (m26b) drains at user-placed points inside long
internal loops. v0 limitation: cells enqueued DURING a
dissolve are leaked.
The C runtime gained the queue surface:
ptr lotus_bus_queue_create(void)void lotus_bus_queue_enqueue(ptr q, ptr handler, ptr self, ptr payload)void lotus_bus_queue_drain(ptr q)void lotus_bus_queue_destroy(ptr q)m20’s “memcpy payload into subscriber’s arena” step happens at ENQUEUE time (publisher’s frame).
m27 + m28a (pinned threads + full lifecycle): Pinned-class
loci spawn a pthread at instantiation; the locus’s full
declared lifecycle (birth → run → drain → dissolve, each only
if declared) executes on that thread, in order. Main thread
continues immediately after spawn. At scope exit (deferred-
dissolve flush), pthread_join blocks until the pinned
thread has finished its lifecycle and returned; the main
thread’s only remaining work for a pinned entry is the join
plus the locus’s arena destroy wholesale (drain / dissolve are
SKIPPED on the main side — they ran on the pinned thread).
m28a synthesizes a per-locus __pinned_main_<LocusName>
function whose signature matches pthread’s start-routine
contract directly (ptr (ptr)); pthread_create gets that
function pointer with self_ptr as its argument. No C-side
adapter, no thread_args struct. The synthesized body simply
calls each declared lifecycle method in sequence, then
returns null.
m28b stage 1 (inline-payload queue): Bus queue cells now
carry an inline [u8; 512] payload buffer (with pthread_mutex_t
guarding the cell array) instead of a pointer to subscriber-arena
memory. The publisher memcpy’s into the cell at enqueue; the
drain (running on the subscriber’s thread) memcpy’s from the
cell into the subscriber’s arena before invoking the handler.
This makes the queue the single point of cross-thread
synchronization: each per-locus arena stays single-threaded
territory, the boundary between layers is where the lock lives.
Per spec/memory.md, “every locus boundary copies the payload”
still holds — just with two memcpy’s per cell instead of one.
m28b stage 2 (cross-thread mailboxes): Each pinned locus
that declares bus subscribe allocates its own
lotus_mailbox_t at instantiation: a bounded ring buffer with
pthread_mutex_t + pthread_cond_t + a shutdown flag, sharing
the same inline-payload cell shape as the global queue. The
locus’s struct grows a __mailbox: ptr field to hold it.
The bus entry table grows from {subject, self, handler} to
{subject, self, handler, mailbox}. Cooperative subscribers
register with mailbox = NULL; pinned subscribers register
with their mailbox pointer. At dispatch time, the bus_dispatch
fn loads entry.mailbox and branches: null → enqueue on the
global cooperative queue (handler runs on the cooperative
thread); non-null → lotus_mailbox_post on the pinned
subscriber’s mailbox (handler runs on the pinned thread).
The synthesized __pinned_main_<Locus> body grows a mailbox
loop between run() and drain(): it calls
lotus_mailbox_drain_one, which blocks on the condvar until
either a cell arrives (returns 1, after dispatching the
handler) or shutdown is signaled with empty queue (returns 0,
breaking the loop). Pending cells flush before the loop
returns 0 even after shutdown — the order check is “queue
empty AND shutdown.”
Coordinated shutdown: at the deferred-dissolve flush, the main
thread calls lotus_mailbox_shutdown on the pinned locus’s
mailbox (sets the flag + broadcasts the condvar), then
pthread_join. The pinned thread observes the empty+shutdown
condition, breaks its loop, runs drain() and dissolve(),
and exits — main joins, then destroys the mailbox and the
arena.
Per The Design / lotus, this is the canonical “any → pinned” bus path: publisher and subscriber sit in different layers of the lotus, the substrate cost lives at the layer boundary (the mailbox lock + the inline payload’s two memcpy’s), and each arena stays single-threaded territory. Bimodality holds.
Still gated: pinned loci cannot declare accept() (children
of pinned would need cross-thread cascade-dissolve
coordination, which is meaningful new infrastructure beyond
m28b’s mailbox post-and-continue) or closures (cross-thread
violation routing). Codegen errors clearly if those are
present.
m28c (CPU-core affinity): When a pinned locus declares
: schedule pinned(core = N), codegen emits a call to
lotus_set_core_affinity(tid, N) immediately after
pthread_create succeeds. The C-side helper wraps
pthread_setaffinity_np (with a cpu_set_t zeroed and bit N
set) so codegen doesn’t have to know the cpu_set_t layout
(opaque + size-variable across glibc versions). Best-effort
semantics: if the requested core is unavailable (e.g., CI box
with fewer cores than the source declares) or the syscall is
denied, the runtime silently falls back to ordinary OS
scheduling rather than refusing to run the binary. The
underlying bimodality is unchanged — pinned(core = N) is a
refinement WITHIN the pinned mode, not a third position.
Topology Phase 1a (cpuset affinity):
pinned(cores = A..B) / pinned(cores = A..=B) /
pinned(cores = {a, b, c}) generalize the single core to a
core set: the thread’s affinity mask is the whole set and
the OS schedules it freely within it — a range carves out an
isolation domain rather than picking one CPU. Bounds are
integer literals (placement is a closed-world deployment
seam), so the compiler expands the spec statically — sorted,
deduplicated — into a constant array and emits one call to
lotus_set_core_affinity_set(tid, cores, count) after
pthread_create. Range inclusivity follows expression
ranges: .. excludes the upper bound, ..= includes it. The
typechecker rejects a spec that selects no cores (4..4,
8..=4) and a duplicated set element; whether the cores
exist on the deploy box stays best-effort at runtime — the
C helper skips out-of-range indices and applies the mask only
if at least one valid core remains. CPU affinity is
Linux-only: on other hosts (macOS) both helpers are compiled
as no-ops and the loci run unpinned. pinned(core = N)
continues to route through the single-core helper unchanged.
Linker dependency: clang invocation now passes -lpthread
unconditionally; small fixed cost in the resulting binary
(libpthread is on every modern Linux).
Bus message router
Section titled “Bus message router”The runtime’s bus is transport-agnostic. From the
framework’s perspective, a transport is the bus kernel projected
through a parameter regime: NATS and UDP multicast and TCP and
Unix sockets are the same primitive (typed pub-sub) at different
(B, c, σ, φ) values. The runtime knows about subjects, channels,
and modes; specific transports come from stdlib (std::bus::*).
- Subject → handler dispatch. Declared
bus subscribe "..." as fndeclarations are wired by the runtime at startup; inbound messages on declared subjects route to the declared handler. - Outbound publish. Declared
bus publish "..."allows emit from any handler return; the runtime routes to the configured transport. - Multi-transport dispatch. A single binary may bind different channels to different transports (a real-time event channel to UDP multicast; a control channel to NATS; a test channel to in-memory). The router maintains per-channel transport bindings established at deployment time from config.
- Transport adaptation interface. Cross-host transports
(NATS, MQTT, TCP-with-framing, custom) plug in via
interface std::bus::Adapter— a contract for user-supplied loci that ship messages on whatever protocol they choose. The contract definition lives inruntime/stdlib/bus.hl; concrete adapter implementations live in user code or downstream packages, NOT in std. The substrate-providedunix(...)transport is in the runtime itself (substrate-guaranteed atomic delivery via SOCK_SEQPACKET) and doesn’t go through the Adapter interface.
v1.x source surface. Subjects are now declared as typed
top-level topic Foo { payload: T; subject: "..."; } decls
(with optional : Parent for hierarchical wire subjects);
deployment-time bindings live in the main locus’s
bindings { Topic: <transport>; } block. Two transport shapes
ship: substrate-provided unix("/path", role: ...) and
user-supplied adapter loci named directly on the right-hand
side (any locus satisfying __StdBusAdapter —
fn send(subject: String, bytes: Bytes) — qualifies).
In-memory delivery is absence-of-entry. Adapter bindings let
protocol-layer transports (NATS, MQTT, TCP-with-framing,
custom JSON-over-WebSocket) live in user code without the
language having to enumerate protocol variants. See
spec/semantics.md “Topic declarations → Phase 2” for the
full surface, including the closed-world topology
optimization that elides bus dispatch for unambiguous
intra-locus and single-hop parent→child tower patterns when no
binding is declared.
Binding realization failure (GH #227) + transports-as-loci
(GH #233 steps 1–2, 2026-07-22). Source-level
bindings { T: unix(...) } entries are sugar: codegen
instantiates a stdlib transport locus
(__StdBusUnixListenTransport / __StdBusUnixConnectTransport,
runtime/stdlib/bus.hl) as a cooperative child at the main
prelude — converging with the adapter path. The locus is the
control plane; the data plane stays in C:
birth()callslotus_bus_transport_realize(subject, path, role)synchronously on the boot path (unixsocket + bind + listenvialotus_transport_listener_createfor listen, connect-with-retry for connect). Realization failure routes intolotus_bus_binding_fail(subject, url)— the structural-failure shape (stderr diagnostic +exit(1)), the same seat aslotus_root_panic. The listen locus’s birth then spawns the serve thread (lotus_bus_transport_spawn_server). The transport loci are deliberately NOT pinned: a pinned locus runs birth on its spawned thread, which would make realization asynchronous; the serve thread belongs to the C data plane.- The serve loop (
lotus_bus_unix_serve) accepts a peer, dispatches its messages, and on peer EOF re-arms — closes the dead connection and loops back intoaccept()for the next peer (GH #233 step 2; peer EOF is not connection loss, and a rolling restart of the connect-side binary just works). dissolve()callslotus_bus_transport_reclaim: sets the serve loop’sclosingflag, shuts down both fds to unblock a parked accept/recv, joins the serve thread, destroys the transport. The husk entry stays in the remote table forlotus_bus_remote_destroy_allto free uniformly.
Publish fanout is untouched — realized entries land in the same
g_bus_remote_entries table the fanout walks.
Connection-loss supervision (GH #233 steps 3–4). Publish
fanout marks a locus-served connect entry lost on send
failure (skipping it thereafter — down-window publishes drop,
never falsely succeed) and pushes it onto a mutex’d pending
list (lotus_bus_transport_mark_lost). The top of
lotus_bus_queue_drain — owner thread, the only place failure
handlers may run — drains the list
(lotus_bus_drain_lost_transports): each handle goes to the
codegen-registered dispatcher (lotus_bus_set_loss_handler,
emitted only when main declares the matching on_failure) or
straight to the structural exit
(lotus_bus_transport_lost_fallback). The dispatcher invokes
main.on_failure(main_self, transport_self, link_lost_violation)
and, when the handler bumped __restart_count via
restart (t), calls lotus_bus_transport_reconnect (re-runs
connect-with-retry against the entry’s stored path; success
clears lost). Codegen support: lotus.main.self global
(stored at main-locus instantiation),
lotus_bus_transport_bind_self (entry ↔ locus self, emitted
after each connect instantiation).
Env-configured routes (LOTUS_BUS_CONFIG) have no source-level
declaration to hang a locus on and keep the direct C path:
lotus_bus_register_remote(subject, url, role) returns an i32
status (0 ok / -1 unrealizable — scheme/addr validation,
socket/bind/listen, connect-retry timeout, thread spawn), with
listener-side work done synchronously at registration (udp:
parse + bind + multicast join via
lotus_bus_udp_listener_setup). On failure the entry is popped
(no dead slot for fanout to silently skip) and
lotus_bus_load_config routes into lotus_bus_binding_fail.
Config-route unix listeners share lotus_bus_unix_serve, so
they re-arm identically. Normative contract: spec/semantics.md,
“The publish contract”.
Adapter dispatch. At codegen, an adapter binding
instantiates the adapter locus into the program-lifetime
payload arena (same m90 routing the -> LocusRef(L) return
path uses), resolves the locus’s send method’s fn pointer,
and registers the (self, send_fn) pair with the runtime via
lotus_bus_register_remote_adapter. The runtime stores both
in lotus_bus_remote_entry_t’s adapter slot. Outbound fanout
packages the wire bytes as an Hale-level Bytes value
(built via lotus_bytes_from_buf against the lazy global
payload arena) and indirect-calls
send_fn(self, subject, bytes). No vtable lookup is needed
at the runtime layer — codegen resolved the method at
binding-emit time.
Adapter inbound (m105). Adapters receiving wire-bytes
from their protocol layer call std::bus::__local_dispatch( subject, bytes); the primitive backs onto
lotus_bus_dispatch_wire, which looks up the subject’s
registered deserialize fn in g_bus_entries (same table the
publish-side fanout consults), reconstructs the struct-layout
bytes, and fans into local subscribers via
lotus_bus_local_dispatch. Symmetric to the unix reader-
thread path; out-of-band recv loops (any code holding wire
bytes for a bound subject) can use this too.
SPSC observation ring (GH #244). A single-producer
fixed-slot ring over CALLER-PROVIDED memory, exposed as lotus
primitives (lotus_spsc_init / _emit / _note_drop /
_set_tag_b / _read) and the raw all-Int Hale surface
std::ring::__spsc_*. Built for observation planes (the iris
observer attaches to these rings inside an shm segment,
read-only, from a foreign process), so the layout is a STABLE
documented contract:
- Descriptor, 64 B, 64-aligned, caller-placed:
u64 data_off(slot array offset from the segment base),u64 head(producer cursor — monotonic, never wraps, published with a release store; slot index ishead & (ring_slots-1), ring_slots a power of two),u64 dropped(producer-side drop accounting; the ring itself never blocks — overwrite-oldest by construction),u32 tag_a,u32 tag_b(user tags; tag_b has a relaxed-store setter for gauge use), 32 B reserved (zero). - Slots:
ring_slots × 16 B, two u64 words, written plain after a RELEASE FENCE and before the head release-store. The fence pairs with an acquire fence on the read side (below) — the Boehm seqlock recipe. Without the pair, a relaxed slot load may observe a future record while the h2 re-read returns a stale head, delivering a mixed record past the discard; GenMC exhibits it (the stress soak cannot — TSO masks it). - Read side (any process, no Hale runtime required): snapshot
h1 (acquire) → copy
[cursor, min(h1, cursor+max))→ ACQUIRE FENCE → re-read h2 (acquire) → discard records with index<= h2 - ring_slotsand count them as overruns. The<=is load-bearing: the producer’s in-flight (unpublished) write for recordhis already clobbering slot indexh - ring_slots, so the live window given a published headhis(h - ring_slots, h]. Verified concurrently intests/spsc_driver.cand modeled inverification/spsc_ring_model.c(which also refutes the strict-<boundary).
Consumer cursors and overrun counters live OUTSIDE the shared
segment (caller-owned); external readers never write the ring.
This ring is the convergence target for iris’s observation
protocol (its PROTOCOL.md pre-freeze sketch is this layout with
tag_a/tag_b as sched_id/current_locus).
SHM ring substrate (Form K5). POSIX shared-
memory ring backing the zero-copy bus route. Six C primitives in
runtime/lotus_shm_ring.c, linked unconditionally so user
programs that bind a topic to a zero_copy route resolve cleanly:
lotus_shm_ring_open(name, slot_size, slot_count)— open or attach (creates if it doesn’t exist; validates header on attach). Returns a per-process handle.lotus_shm_ring_claim(ring)— publisher gets a pointer to the next slot. v1 is single-producer; never fails.lotus_shm_ring_commit(ring)— release-orders the slot writes before atomic-incrementing the published seqno.lotus_shm_ring_published(ring)— acquire-load the seqno for subscriber-side polling.lotus_shm_ring_read_slot(ring, seqno)— subscriber gets a pointer to the slot forseqno. Returns NULL if not yet committed OR wrapped past (slow consumer).lotus_shm_ring_close(ring)— unmap + close fd; unlink the SHM object if this handle created it.
Layout in SHM: a 64-byte cache-aligned header
(lotus_shm_ring_header_t — magic, slot_size, slot_count,
atomic seqno) followed by N slots of slot_size bytes each.
Header magic + sizes are validated on attach to catch ABI
mismatches across binaries pinned to the same ring name.
Foreign-layout consumer (Proposal B). Two more
primitives read an externally-defined ring described by a
ring_layout (see semantics.md § “Foreign rings”), rather than
the native LRSRNG1 shape:
lotus_shm_ring_open_layout(name, desc)— attach an existing foreign segment READ-ONLY (never creates),fstatfor the map length, validatemagic/version, readbuffer_sizefor the data-region capacity.descis alotus_shm_layout_tbuilt from a flat 16-entry uint64 descriptor codegen emits.lotus_bus_register_subscriber_shm_ring_layout(subject, name, desc_words, self, handler)— open via the above and spawn abyte_recordsreader thread that walks[len_prefix][payload]records (modular overcapacity, skippingpad_sentineltail-pads, advancing byalign_up). Shares the native subscriber registry +atexitteardown; a layout subscriber is markedis_layoutand torn down vialotus_shm_ring_close_layout.
The producer side (Proposal B M3a) mirrors these:
lotus_shm_ring_create_layout(name, desc, capacity)— CREATE + own the segment (sizedata_at + capacity, write the magic/version/buffer_sizeheader, zero the cursor). Attaches read-write without re-init if it already exists.lotus_bus_register_shm_ring_layout(subject, name, desc_words, capacity)— create via the above + register a producer (one per subject).lotus_bus_publish_shm_ring_layout(subject, value, size)frames onebyte_recordsrecord (the inverse of the reader: reservealign_up,pad_sentinelat the wrap, write the length prefix + payload, release-store the cursor). The producer rings are closed +shm_unlink’d atatexit.
Field reads/writes are host-native endianness (the foreign producer and Hale are
both little-endian x86-64). v1 is byte_records only; the slots
framing kind and a zero-copy writable producer view are post-v1.
v1 scope: single-producer, multi-consumer; in-memory delivery is
in scope (POSIX shm_open works intra-machine cross-process).
Multi-producer (CAS-based claim), back-pressure / timeout
modes, and named-ring registry are post-v1. The Hale-side
fallible(ClaimError) signature is reserved for those; v1’s
claim() never actually fails.
Lifecycle / cleanup. Both
lotus_bus_register_shm_ring (publisher) and
lotus_bus_register_subscriber_shm_ring (subscriber) register
a single atexit hook on first call. The hook:
- Signals every subscriber reader thread to stop via an
atomic
should_stopflag. pthread_joins each reader thread, ensuring no in-flight handler is interrupted.- Frees the subscriber state allocated by the registration call.
lotus_shm_ring_closes every ring opened in this process — whichshm_unlinks the ones this process created (owns_unlink=1), keeping/dev/shm/clean across restarts.
The atexit hook runs on a clean process exit (return from
main, exit(3)). Signal-driven termination (SIGTERM, SIGKILL,
_exit) bypasses atexit and leaves the SHM namespace entry
behind until reboot or manual shm_unlink. A future v1.x
SIGINT/SIGTERM handler can fold into this same teardown.
Constraint: subscriber handlers must not call exit(). The
handler runs on the reader thread; calling exit() from inside
the handler invokes atexit on the reader thread, which then
attempts to pthread_join itself (undefined behavior). Use
_exit() if a handler needs to terminate the process
immediately.
Codegen surface for K5 lands in K4 (route-selection + slot-locus synthesis) and K6 (subscriber view + epoch guard). K5 ships the substrate; user code can’t reach these symbols directly without going through the slot-locus surface a zero_copy binding produces.
Closure-test infrastructure
Section titled “Closure-test infrastructure”- Default epoch is
dissolve. Closures with noepochclause evaluate at the locus’s dissolution. Other epochs:epoch tick,epoch duration(...),epoch birth,epoch explicit— runtime-managed per declaration. - Accumulator engine. For each
closure name { ... }, the runtime maintains accumulators for the left and right sides of~~, scoped per epoch (when accumulation is needed; not needed for one-shot self-referential closures likeself.x ~~ self.y within 0). - Band checking + reporting. At each epoch boundary, the
runtime evaluates left and right expressions, checks the
band, and emits a typed
ClosureReportevent the application can subscribe to via bus. - Collapse vs. explosion. A closure-pass at any epoch is
silent. A closure-fail flips an “exploded” flag on the locus.
At dissolve, if exploded, the parent’s
on_failure(self, ClosureViolation { ... })is invoked with a typed event carrying closure name, epoch, left/right values, tolerance, diff. Distinct from hard substrate failures (OOM, divide-by-zero, null-deref from miscompilation) — those terminate the process directly without the ClosureViolation routing path. See decisions §F.9. - Recovery-event interaction.
persists_through(...)andresets_on(...)clauses are honored at recovery time; the accumulator is preserved or zeroed per declaration. The exploded flag itself persists acrossrestart_in_placeandquarantine(per default; futureclear_violation_on(...)clause may override).
Perspective infrastructure
Section titled “Perspective infrastructure”- The global slot. Each
perspective Phas one program-global{ data, vtable }slot (__persp.<P>). Every holder ofperspective(P)dispatches through it — a load plus a predicted indirect call, near-direct cost. A program that declares no perspectives pays nothing. - Live swap (
reperspective). Re-points the slot at a newserves Pimpl with a single atomic store, redirecting every call site at once. State-preserving across impls of one footprint: the{ data, vtable }split meansdata— the live, arena-backed state — is untouched and only the vtable changes. When the perspective declares a bus surface, the swap also re-points its subscriptions on that samedata. (Seespec/semantics.md§ Perspectives.) - Wire hot-load (aspirational). Transport-driven redeploy —
decode a serialized perspective against the compiled-in schema,
gate on
stable_when, atomically install with no torn read — is specified but not yet shipped. Seespec/semantics.md§ “Perspective hot-load”.
Failure handling
Section titled “Failure handling”- Failure =
ClosureViolationpropagation. Anyclosureassertion that fails in a locus body produces aClosureViolationrecord routed to the parent’son_failure(child, err)handler per F.9. The parent picks one ofrestart/restart_in_place/quarantine/reorganize/bubble, or absorbs (returns without calling any). A violation that bubbles past the root exits the process non-zero with the violation report on stderr. - No source-level panic / exceptions. Hale has no
panic(msg),assert(cond),throw/catch, or implicitly-propagating exception machinery. Failure is either structural (closure violation; parent-policy recovery, Erlang let-it-crash with the parent locus as the supervisor) or value-level viafallible(E)(v1.x-FORM-1 addressing protocol; every fallible call MUST be addressed by anorclause at the immediate caller). The two channels are orthogonal at every frame except the implicit main locus’s root, where a value error escaping past every enclosingfallibleframe triggerslotus_root_panic— the runtime’s only value-error escape valve. Seespec/semantics.md§ “Process exit”.
Native observation emission (iris P4, 2026-07-27)
Section titled “Native observation emission (iris P4, 2026-07-27)”With LOTUS_OBS=1 the runtime publishes an iris-protocol
observation segment (/hale-obs-<pid> + registration file per
PROTOCOL v0.2 — the contract lives in the iris-observer repo;
layouts mirrored from its reference implementation. 0.2, 2026-08-12
downstream handoff: the header gains model_hash at 0x80 — the
topology artifact’s shape_hash, stamped by the CLI from the same
bundle it typechecks, so a consumer can establish the RUNNING
binary was built from the model it joins against; 0 = unstamped
harness build. And the per-binding backpressure cells reserved
since v0 are now written: queue_depth (cell 3, a last-write-wins
gauge of the kernel send-queue occupancy sampled at send time),
send_block_ns (cell 4, accumulated transport-send duration), and
retries (cell 5, reconnects) — counters-tier, so a consumer
falling behind shows as depth climbing and block time accruing
BEFORE anything drops) and the
runtime’s own choke points emit records: BUS_PUBLISH /
BUS_DELIVER at every dispatch flavor (dynamic, static-devirt,
cross-thread wire; deliver is enqueue-time at v0),
NET_SEND / NET_DELIVER at the transport fanout/reader with
per-binding monotonic seqs (what iris seq-matches into
cross-process edges), LOCUS_BIRTH / LOCUS_DISSOLVE (both
cooperative and pinned lifecycle paths; dissolve rides the
emit_locus_arena_destroy chokepoint) and RESTART at
reconnect. Cost contract: env unset = one predictable branch
per probe (g_obs_state); enabled-but-unobserved
(observer_count == 0) = counters only, no ring writes; ring
emission is SPSC via the #247 primitive with one ring per
emitting thread (TLS assignment; overflow threads count into
ring_drops_total). On the observer-count 0→1 rising edge the
live-locus table replays as LOCUS_BIRTH records so a
late-attaching observer reconstructs the tree. Knobs:
LOTUS_OBS_RINGS (default 8), LOTUS_OBS_SLOTS (default
4096), LOTUS_OBS_WIRE (default off — see NET seq semantics:
LOTUS_OBS alone never alters the wire; the cross-process edge
header rides the wire only when the operator opts the whole fleet
in with LOTUS_OBS_WIRE=1). v0 notes: manifest ids are registration-order; publisher
locus attribution on BUS_PUBLISH is unattributed (0); pinned
births render parent=root. lotus_obs.c is its own TU; the
arena TU’s probes are weak-guarded so helper binaries compiling
lotus_arena.c alone still link.
Field-hardening (iris handoff 2, 2026-07-27, v0.11.13): NET_SEND
now fires on the UDP multicast fanout path (it continues before
the stream-transport probe, so multicast publishers emitted only
deliver-side records — the seq matcher had nothing to pair);
LOCUS_BIRTH is emitted BEFORE a locus’s field-default init and
carries real parentage (the parent registers before its children
look it up — post-birth placement rendered every tree flat), and
pinned children register on the spawning thread so a park before
their first probe doesn’t hide them; BUS_PUBLISH stamps the
publishing locus from a codegen-set TLS (the C dispatch doesn’t
know self); topic shape_hash is fnv(subject, canonical payload structure) — field names + coarse type tags in declared order,
never the declaring type’s local name — so two binaries sharing a
subject fuse into one manifest row; and each ring re-emits EPOCH
every 1024 records so a high-rate ring that wraps its anchor never
reconstructs timestamps from a stale base (the ~2^64 ns readings).
NET seq semantics (iris handoff 3, v0.11.14): NET_SEND and
NET_DELIVER carry w1 = origin:16 | seq:48 where origin is
the SENDER’s per-process identity and seq is the SENDER’s
per-binding counter — the receiver echoes both verbatim from
the wire, so a send and its delivers pair on (origin, seq)
across segments (the cross-process edge). Before this, the
receiver stamped its LOCAL delivery counter, which sums across
senders on a multicast subject — the send seq never equalled the
deliver seq and iris rendered zero edges. (Stream transports are
unicast — one sender per connection — so origin 0 + the framed
wire seq already pairs correctly.)
Edge-emission correctness (iris handoff 4, v0.11.15):
- The record id field IS the topic id for
NET_SEND/NET_DELIVER(as for the bus records) — it is the consumer’s join key onto the fused topic row. It was hardcoded 0, so no NET event could be associated with any topic and edges were structurally impossible regardless of(origin, seq). The probe now resolves the id from the subject (in hand at every emit site); the per-binding counter line still keys off the binding id. - The published counter is never attribution-gated. Counters
are the dormant-mode contract (enabled-but-unobserved = counters
only), so a genuine publish counts even when its locus can’t be
attributed. handoff-3 gated the counter (and record) behind a
positive publisher-TLS that a cross-pool or free-fn publish never
set, zeroing the fleet’s published counter. Inbound wire
re-dispatch is now excluded by NEGATIVE marking instead: the
reader brackets its re-dispatch (
lotus_obs_begin/end_redispatch) and the publish probe consumes the mark; genuine publishes are the unmarked default and always count, with best-effort locus attribution (0 when unknown). The keyed dispatch flavors, which had no publish/deliver probe at all, now emit both. LOTUS_OBSnever alters the wire. The(origin, seq)edge identity is a wire change a pre-header receiver cannot parse — an observed sender would silently drop every datagram at a stale peer, partitioning a mixed-version fleet invisibly. So the UDP self-describing header ([u64 magic][u64 origin|seq]) and the framed-transport origin word ride the wire ONLY when the operator opts in withLOTUS_OBS_WIRE=1; withLOTUS_OBSalone the wire is byte-for-byte identical to an unobserved run. Cross-process edges therefore requireLOTUS_OBS_WIRE=1fleet-wide; counters and local records need onlyLOTUS_OBS=1. The receiver always peels a header it recognizes (self-describing), so old→new and unobserved peers are unaffected either way.
Flavor completeness + quiet-process replay (iris handoff 5, v0.11.18):
- The fully-devirtualized direct dispatch now carries probes.
The single-quiet-subscriber same-thread flavor (the
static_directbucket walk with the handler baked as a direct call, and its multi-handler C sibling) emitted NO probes at all — a subject on that path never registered its topic, counted nothing, and produced no BUS records. Both direct flavors now publish once + deliver per matched target like every other flavor. Dormant cost is preserved by thelotus_obs_livegate: codegen checks the flag ONCE per function entry (the flag is final before any user publish can run — obs resolves at the first probe, always a locus birth), so LLVM hoists the branch and an unobserved publish loop is instruction-identical to the probe-free lowering. The same per-fn check now also gates the publisher-attribution TLS note. The AST-level rewrite got the same sentence on 2026-08-11 (downstream handoff P23): the closed-world intra-locus/tower desugar — one rewrite earlier than these codegen flavors, so the earlier fix never reached it — now emitsBUS_PUBLISH(publisher-attributed) +BUS_DELIVER(subscriber-attributed) at the direct handler call, behind the same per-fn gate. Its first probe registers the topic’s manifest row, so an intra-tree subject with traffic is indistinguishable from its bus-dispatched sibling, and manifest absence uniformly means “never mentioned at runtime” on every flavor. - Observer attach no longer needs probe traffic. The 0→1
birth replay was driven from inside probes, so a probe-quiet
process (a main parked in a read loop with pinned raw-fd
readers; hot paths on the direct flavor) never noticed an
observer and stayed silent — segment registered, zero records.
When
LOTUS_OBS=1, a detached heartbeat thread drives the replay check every 250ms under the obs lock (teardown takes the same lock around the unmap, so the heartbeat never races it); replay latency after attach is bounded at ~250ms even for a process that never probes again.
BUS record w1 layout (iris handoff 7, v0.11.21): BUS_PUBLISH /
BUS_DELIVER pack w1 = locus:20 (bits 44..63) | seq:44 (low) —
PROTOCOL §8, executable reference iris emitter/protocol.h
(obs_bus_w1). The emitter had these fields transposed (locus
low, seq shifted) since handoff-3; attribution was computed
correctly and packed unreadably, and the contract tests stayed
green because they decoded with the emitter’s own layout. The
tests now vendor protocol.h’s decode, so emitter and consumers
cannot disagree silently.
Adapter ingest completeness (iris handoff 8, v0.11.22): the
Hale-owned-wire ingest (std::bus::__local_dispatch →
lotus_bus_dispatch_wire_inbound) now carries the full probe trio
the C reader threads have. The wrapper peels the self-describing
obs wire header when the producer emitted one (magic-guarded — a
headerless or non-Hale producer is byte-for-byte unaffected, and
headered bytes reaching a Hale adapter previously failed
deserialization outright), emits NET_DELIVER echoing the wire
(origin, seq) (or (0, local per-subject seq) when headerless —
countable, not pairable), and the plain dispatch_wire fanout
gains the per-target BUS_DELIVER its keyed sibling received in
v0.11.18. Adapter bindings register lazily in the manifest
(aux = 2). Dynamically-ingested planes now form producer→consumer
edges and attribute subscriber loci exactly like statically
configured listens.
Attribution ordering + the adapter inbound path (iris handoff 6, v0.11.20):
lotus_obs_liveis resolved in a constructor, beforemain. The fn-entry gate hoist’s soundness argument requires the flag to be final before ANY function entry — previously it was set at the first probe (a locus birth inside main’s body), so a publish lowered intofn mainitself would snapshot a stale dormant flag forever. Constructor resolution makes the flag genuinely process-constant. The field-shaped ordering case (publishers deep in steady-state loops, observer attaching much later) is pinned inobs_fleet_contract.rs.std::bus::__local_dispatch(the adapter-inbound ingest) is redispatch-MARKED. An adapter locus dispatching received wire bytes is a delivery, exactly like the reader-thread path — the unmarked entry stamped a spuriouslocus=0BUS_PUBLISH per inbound message and inflated the published counter, so a fleet ingesting via the std adapter surface read as “everything publishes unattributed.” Lowers tolotus_bus_dispatch_wire_inbound, which brackets the dispatch with the P15 negative marking. Per-subscriber BUS_DELIVER and the topic registration are unchanged.
Lossless recording mode (GH #296 Phase 1)
Section titled “Lossless recording mode (GH #296 Phase 1)”LOTUS_OBS_RECORD=<path> opts a run into recording: the
observation plane stops being a sampler and becomes a flight
recorder. Implies LOTUS_OBS=1; the recording counts as an
attached observer, so ring emission is on from the first probe
with nothing mmapping the segment. Opt-in on the same terms as
everything else here — unset, the lowering and the wire are
byte-for-byte the unobserved build (the flag resolves in the same
constructor as lotus_obs_live and gates behind it everywhere).
Three deltas from observer mode, all in the service of one rule — a recording never drops (a dropped record is a replay that diverges silently):
- Overwrite-oldest becomes block-the-producer. An in-process
drain thread appends every ring record to
<path>and publishes per-ring cursors; a producer whose ring is full against the cursor waits (50µs naps) instead of clobbering the oldest slot. Recording-on hot cost while the ring has room is one relaxed load + compare per record. A live observer prefers losing records to stalling the program; a recorder makes the opposite trade, and the stall is the contract. - A thread that cannot get a ring fails the run (exit 70,
diagnostic names
LOTUS_OBS_RINGS) instead of going silently dark forever (the-2disposition observer mode keeps). Recording defaultsLOTUS_OBS_RINGSto the 64 maximum, so this means more than 64 emitting threads. Block or fail — never drop. - A drain-side write failure fails the run (exit 74) rather than truncating silently. The file ends with a trailer written at teardown after a final full sweep; a reader that does not find the trailer holds a truncated recording and must say so.
Durable recording / crash-prefix recovery (Phase 5). Named
precisely: this is a durable flight recorder, NOT a write-ahead
log — the application is never gated on the recording reaching
stable storage, so a crash can lose records the application
already executed past. (A load-bearing WAL grade — persist,
fence, then permit delivery — is a possible later mode; nothing
here claims it.) What IS guaranteed: the drain appends whole
frames in stream order and flushes every sweep, so the on-disk
file is an exact prefix of the stream at all times, and the
header identity (model_hash, exec_digest, policy flags) is
stamped eagerly by the drain as soon as the ctor-sequenced
setters have run, not only at finalize. A run that dies without
teardown therefore leaves an artifact that is attributable and
exact up to one torn frame at the tail. Default recording rides
the page cache (survives a process crash, not power loss).
LOTUS_OBS_RECORD_DURABLE=1 is the power-loss grade: every
flushed sweep passes fdatasync, the parent directory is synced
at file creation (a fresh NAME is not durable until its directory
entry is), the finalize trailer itself is fdatasync’d before
close (a clean exit followed by power loss must not demote a
finalized recording to a truncated one), and the grade is
recorded in the header’s policy flags (bit 1).
Consume records (private recorder namespace). BUS_DELIVER
is enqueue-time by design and lands on the publisher’s ring —
correct for fanout accounting, structurally unable to say in what
order a consumer actually ran its handlers. Under recording, every
dequeue-driven handler invoke (main-queue drain, coop-pool drain,
pinned mailbox drain, async coro start) stamps a consume record on
the consuming thread right before the handler runs, carrying the
delivery identity: the target locus in the record’s id field and
the full 64-bit msg_id in w1. Its ring position is the
per-consumer delivery order — the thing a replay serves back.
Pairing rule: per queue, the k-th consume is the k-th queued
delivery. The synchronous direct-dispatch flavors deliberately emit
no consume record: their handler runs at the BUS_DELIVER
position, which is the consumption point.
Recorder events (CONSUMER/CONSUME/ENQ/JOURNAL) live on
process-private per-thread rings in their own event namespace,
never in the public observation segment: iris protocol ekinds
8/9/11/12 mean SUPERV_TRANS/PLACEMENT/BINDING_UP/BINDING_DOWN, and
an observer attaching to a recording process must never decode
recorder bookkeeping as lifecycle transitions. In the recording
file, private-ring entries carry the high bit in their ring field,
so the two namespaces can never be confused.
Delivery identity (Phase 2). Every queued delivery carries a
deterministic pub_id = consumer_id:16 | per-publisher-thread seq:48 (full width — the review-round guards fail loudly rather
than wrap), re-derivable by a re-executed run without global
coordination. Consumer ids are stable across runs, unlike pthread
ids: main = 1, cooperative pool workers = 16 + registration
index, pinned locus threads = 64 + obs instance id; threads with
no stable identity (ingress readers) get run-unique anonymous
ids, never a shared fallback. Payload bytes are captured once per
queued publish under a stable subject hash (manifest topic
ids are registration-order and racing publishers register in
either order). Flags: bit 0 = external wire ingress; bit 1 = raw
in-process struct bytes — an ABI snapshot (String/Bytes fields
are pointers, padding is uninitialized) that consumers must
compare by size only; canonical per-topic recording codecs are
the staged fix. Wire captures are canonical bytes, unflagged.
The synchronous direct-dispatch flavors deliberately capture
nothing: a closed-world same-thread call cannot carry external
input, and re-execution re-derives its payloads.
Input journal (Phase 3). Under recording, every user-facing
nondeterministic read is journaled per consumer as ONE unified
stream carrying the read’s kind and its exact encoded
arguments (length-framed; replay memcmps them — hashes proved
collision-prone), so a changed env name, arg index, rand bound,
or read length is the first named divergence, never a silently
substituted value. Env values are withheld by default: names,
existence, and lengths are recorded; the value itself requires
LOTUS_OBS_RECORD_ENV=full, the artifact header says which
policy applied, and a withheld read replays as a named withheld
divergence. The interposed set:
std::time::now, std::time::monotonic[_ns] (now a named
primitive, lotus_time_monotonic_ns — it used to be inline
clock_gettime IR that nothing could interpose),
std::rand::next_int (per call — the global RNG’s mutex makes
seed-replay unsound across threads), std::os::getrandom, and
the std::env surface. Internal runtime clocks and config
getenvs are deliberately not journaled.
File format v0.3 (PRE-STABLE). 96-byte header (magic
HALEREC0, version, pid, ring geometry, epoch anchors,
model_hash, a 32-byte framed-SHA-256 exec_digest, and policy
flags), tagged entries — tag 0: 24-byte ring record; tags 1/2/3:
payload / journal / meta blobs (32-byte header + bytes padded to
8; meta carries the run-stable topic and public-ring identity
maps) — and a 16-byte trailer (HALEEND0 + entry count). A
recording is clean only when the ENTIRE artifact validates:
exact parse to the trailer position and a matching entry count,
enforced by the CLI reader and independently by the runtime
loader (one open + fstat + private mmap, checked arithmetic,
per-kind value shapes) — trailer magic at EOF alone proves
nothing. Identity fields are stamped eagerly by the drain and
re-stamped at finalize (a prelude binding registration can probe —
creating the header — before the stamps run; the two stamps
agree). A trailer-less artifact is a crash-truncated
recording: refused by default, but admissible as a PREFIX with
hale replay --allow-truncated (LOTUS_REPLAY_ALLOW_TRUNCATED=1)
— the loaders stop at the first incomplete frame, report the
parsed extent and dropped tail bytes, and replay that prefix;
execution past the tape’s end falls back live and is counted, per
the degrade-never-refuse rule. Under --diff a truncated baseline
compares as a prefix — and the runtime verdict agrees with the
comparator: recorded-history-exhausted events (journal reads past
the tape, deliveries past the consume stream, consumers the
recording never saw) count into a separate
post_prefix_live_fallback status key, never into the divergence
totals; mismatches BEFORE exhaustion stay divergences either way.
A trailer-FINALIZED artifact keeps the exact full-parse + count
checks: with a finalize present, any truncation is corruption.
Replay (hale replay <recording> <program.hl>). Admission,
strongest check first: the recording’s exec_digest — a framed
SHA-256 over the toolchain source hash (compiler + runtime +
stdlib implementation, via the stale-CLI build hash), the CLI
version, build options, and every source file’s full path,
length, and contents — must match the recompiled program exactly;
shape_hash is structural compatibility only, the secondary
check. An unstamped recording is refused without
--allow-unverified-model. Stated residue: the digest cannot see
the LLVM/libc toolchain outside the hale binary; a post-link
binary digest is the staged stronger form. Safe by default:
replay re-executes real side effects, so the typed effect rows
gate admission and fail CLOSED on syscall, ffi,
unclassified (“may do anything”) — refused without an explicit
--allow-live-effects (coarse by class granularity —
over-refusal is the safe direction; per-primitive replay classes
are the staged refinement). bindings blocks are no longer
in that list: under replay the wire is hermetic (below), so a
bound topic cannot reach the live world. LOTUS_REPLAY=<path>
then serves journaled reads back per consumer in recorded order;
a read past (or mismatching) the recorded history falls back live
and is counted — replay degrades, never refuses (RFC Q6) —
with a divergence summary at exit and a machine-readable verdict
(LOTUS_REPLAY_STATUS) the CLI consumes: journal misses, order
holds, unconsumed journal entries, and unconsumed deliveries all
count, and any of them fails --diff. Each consumer’s queued deliveries are re-consumed
in the RECORDED order (Phase 4): dequeued cells that arrive ahead
of their recorded turn are held per-consumer and released in
order, with a bounded hold (1s) after which the oldest held cell
is released and the miss counted, so a genuinely divergent replay
reports rather than deadlocks.
Async pools replay (Phase 6). The nondeterminism of a where async_io pool is its drain’s SCHEDULING: which cell starts when,
which parked coro resumes when (epoll readiness order), and when a
timed park expires relative to both. Under recording, every such
decision is stamped on the pool worker’s private ring as a step
(ASYNC_START/ASYNC_RESUME/ASYNC_EXPIRE; coros are named by
birth ordinal — the index of the START that created them, stable
across runs because start order is enforced). Under replay the
drain is DRIVEN by that stream instead of the clock: START steps
reuse the phase-4 cell gate (start order is consume order); RESUME
steps wait for the named coro’s readiness, parking early-ready
coros aside until their turn; EXPIRE steps resume the named coro
with the timed-out sentinel immediately — the recording already
proves the deadline fired at this point in the sequence, so
replayed sleeps fast-forward rather than re-waiting. A step
unsatisfiable within the hold bound counts an async-schedule
divergence and is skipped (degrade, never deadlock) — and a
skipped START retires both its birth ordinal and its consume
slot (review round 2: ordinals belong to the recorded START slots,
not to whichever cell happened to start next — without retirement
one missing delivery shifts every later coroutine into an earlier
recorded identity, and the pinned consume expectation makes every
later START mismatch; with it, later steps that name a retired
slot skip immediately and the divergence stays local).
Coverage, named (review round). A dry tape hands the pool back
to the live drain — never silently. The artifact carries a
capability bit (header flag 4: this runtime records async
schedules); the states are distinct and reviewable: a pre-phase-6
artifact gets a one-shot coverage note (“recording predates
async-schedule support”); a truncated tape’s remaining schedule is
the stated coverage boundary (excluded from the divergence
totals); a FINALIZED capable artifact whose tape runs dry while
the re-execution keeps doing async work counts each such action as
async_post_tape, and schedule steps left unconsumed at exit
count as unconsumed_async_steps — both machine-readable status
keys and divergence rows. --diff compares the per-consumer async
step streams (kind, value) bidirectionally, exactly like the
consume and journal streams — “byte-identical output” alone is
weaker than “the schedule matched”. Boundary, stated: the SCHEDULE replays; the DATA of
unjournaled I/O does not — a coro resumed at its recorded turn
re-executes its recv against the live world (syscall-class, gated),
and bindings ingress arrives via the Phase-5 injector.
Hermetic wire + ingress injection (Phase 5). Hermeticity is
a binding-kind capability, not a blanket assumption: the
native unix:// and udp:// transports (and the transport-locus
form) are suppressed and injectable; a backend with no replay
class fails closed — shm_ring in particular is refused by the
CLI (backend named) and independently at the runtime’s shm-open
seam, so a replayed or fed process never creates, opens, or
mutates live shared memory. Adapter loci are user logic and
re-execute as such (governed by their inferred effects).
For the covered backends, under replay a binding is suppressed
at realization: the entry exists (a husk, transport NULL —
the same shape the reclaim path already leaves), but no socket is
created, nothing binds, nothing connects, and outbound fanout to
bound subjects sends nothing (the publish is still recorded and
compared under --diff).
In the listeners’ stead, injection runs as an explicit boot
phase: codegen emits one lotus_replay_start_ingress() call at
the main locus’s boot/run boundary — params children born,
bindings realized or suppressed, every boot subscription
registered, run() not yet entered — where the runtime snapshots
the subscription registry ON the main thread (injector workers
never read the live registry) and spawns one worker per
recorded ingress source (the pub_id’s consumer bits), each
carrying its source’s recorded consumer identity so a --diff
verify recording aligns per-consumer streams with the original
instead of collapsing every listener onto one injector identity.
Fresh anonymous claims are floored above the recorded range.
The snapshot IS the coverage boundary: a
subscriber the program only creates during run() (spawned or
accepted) is not visible to injection — such tape entries classify
as late_subscription_uncovered (the injector join rescans the
live registry at teardown to distinguish them from genuinely
absent subscribers), a stated boundary rather than a silent
unmatched. Injection is keyed by full identity: each tape record carries
its complete subject string and the subject’s canonical payload
shape (FNV-32 alone is collision-prone and is kept only for
reporting); a record whose shape does not match the live
program’s shape for that subject is refused as incompatible,
never fed as a plausible wrong value. Only messages the original
application accepted enter the tape — the wire capture runs
after the deserializer says yes, so identity allocation agrees
between record and replay and rejected traffic is never re-fed.
Pacing. Strict replay injects one message, then waits (bounded
by the phase-4 hold) until that message’s recorded consumes have
all been re-consumed — unpaced injection can shed at bounded
queues (bounded(N, drop_old/drop_new), on_full policies)
before the dequeue-side order gate ever sees the cell, which no
amount of holding can undo. Feed mode is deliberately unpaced.
Accounting. Every tape entry ends in exactly one class:
injected, rejected-by-deserializer, unmatched-subject,
incompatible-shape, or unprocessed-at-shutdown (the injector is
joined at teardown and its remainder classified); injector start
failure is its own class. Under strict replay every non-injected
class is a divergence, in the machine-readable status
(ingress_*, injector_start_failure keys) and the exit
summary.
Feed mode (hale replay --feed, LOTUS_REPLAY_FEED=<path>) —
same recorded ingress, changed code, live nondeterministic
environment. Stated that precisely because “same inputs” would
overclaim: feed reproduces recorded binding ingress only —
time, randomness, env, and argv stay live. The recording is
consumed as an input tape: recorded ingress injected, wire
hermetic, and nothing else of replay applies — no journal
serving, no order enforcement, no model admission (a model-hash
mismatch is reported informationally; feeding a tape to changed
code is the point), no --diff/--at (there is no recorded
schedule to compare against). The effects gate still applies:
--feed bypasses identity admission, never effect safety — a
program whose frontier reaches syscall/ffi/unclassified
still requires --allow-live-effects alongside --feed.
Mutually exclusive with LOTUS_REPLAY. Injected deliveries carry
the new run’s own identity. The exit report classifies every tape
entry, and an unfed remainder (unmatched, incompatible,
unprocessed, start failure) fails the run by default —
--allow-unmatched-feed is the explicit acceptance of a partial
feed.
--diff (strict replay only — feed rejects it, above).
--diff records the replay (under the original’s env policy) and
compares bidirectionally: per-consumer queued consume streams
(target locus + msg_id), per-consumer PUBLIC bus streams aligned
by subject and stable consumer id via the meta maps — which is
what makes synchronous direct dispatch visible — payloads both
ways (topic, flags, bytes; raw metadata by declared size), and
per-consumer journal streams (kind, exact args, withheld state,
value; grouped per consumer because the drain interleaves
concurrent threads’ entries in incidental order). Any runtime
divergence fails --diff through the verdict. --at <n> stops
(SIGSTOP) at the nth consume process-wide (meaningful for one
consumer); --at <consumer-id>:<ordinal> is the stable
multi-consumer form. Replay implies observation (identity rides
the obs machinery).
Two honest limits, stated rather than implied: the recorded interleaving is reproduced per consumer, not globally — cross- consumer wall-clock alignment is not a replay property; and the injected tape covers bindings ingress — raw user-level socket reads are syscall-class live effects (gated), and adapter loci re-execute their own protocol logic. Fleet-scale replay (multiple binaries against one composed tape, with #262’s plan admission for replay-under-a-different-plan) is the next milestone.
- Monotonic + wall-clock.
time::now()andtime::monotonic()are runtime-provided.time::monotonic()returns aDuration(i64 nanoseconds since an unspecified reference); only meaningful for elapsed-time differences. Backed byclock_gettime(CLOCK_MONOTONIC).time::now()(C7, pond follow-up) returns wall-clock seconds since the Unix epoch asIntviaclock_gettime(CLOCK_REALTIME); observation only — NTP slewing and leap seconds can warp the value, sotime::monotonicstays the basis for scheduling. RicherTime-typed wall-clock (with calendar arithmetic) is deferred until a consumer surfaces a concrete date-shape need. Mocking is available for tests viatime::mock_clock(...)(stdlib). - Monotonic-only scheduling. Every scheduling primitive in
Hale —
time::sleep,time::tick, the cooperative scheduler’s deadline queue — is grounded on the monotonic clock. NTP slewing, leap seconds, and wall-clock jumps cannot warp scheduling decisions.time::sleepretries on EINTR using the kernel’s reported remaining time, so a delivered signal does not shorten the total sleep.CLOCK_REALTIMEis used bytime::now()for wall-clock observation only and has no scheduling role. - Implementation invariant.
time::sleep(d)lowers toclock_nanosleep(CLOCK_MONOTONIC, 0, &req, &rem)with EINTR retry — important for a system targeting high-precision clock semantics.
I/O — minimal
Section titled “I/O — minimal”- stdout / stderr for
print/println. That’s it for runtime-level I/O. Files, networking, etc. live in stdlib. - Errno surface helpers (2026-05-16, used by the fallible
std::io::fs::*/std::io::tcp::*wrappers):lotus_get_errno() -> i32— surfaces the current platformerrnoto LLVM. Each fallible wrapper calls this immediately after the failing primitive (POSIX errno is sticky until the next syscall sets it).lotus_io_error_kind(errno_val: i32) -> *const char— maps errno to a stable kind-tag string ("not_found","permission_denied","is_dir","already_exists","would_block","connection_refused","timeout","host_unreachable","broken_pipe","interrupted", …, catch-all"io"). Returns a static-table pointer; caller must not free.
Text + string primitives (v1.x adds)
Section titled “Text + string primitives (v1.x adds)”-
lotus_str_parse_float(s) -> double/lotus_str_can_parse_float(s) -> int— v1.x-16. Strict trailing-NUL parse; 0.0 on failure paired with a bool predicate. Mirrors the parse_int contract. -
lotus_text_base64_decode(s) -> Bytes*— v1.x-16. Standard alphabet, whitespace tolerated, non-alphabet / wrong padding returns empty Bytes. Inverse oflotus_text_base64_encode. -
lotus_str_builder_new()/_append(b, s)/_len(b) -> i64/_finish(b) -> char*— v1.x-15. Doubling-realloc malloc buffer. N appends are amortized O(N).finish()copies into the bus payload arena (program-lifetime) and frees the builder. -
lotus_bytes_builder_new(i64 initial_cap) -> ptr/_append(ptr handle, ptr chunk) -> i64 status/_len(ptr handle) -> i64/_finish(ptr handle) -> Bytes*/_shift_front(ptr handle, i64 n)/_clear(ptr handle)/_snapshot(ptr handle) -> Bytes*/_view(ptr handle) -> Bytes*/_free(ptr handle)— C10 / Phase 0 / Phase-2 (1) (2026-05-19, pond/websocket follow-up). Binary-safe sibling of the str-builder family. Append reads the chunk’s[i64 len]prefix instead ofstrlen; finish emits a length-prefixed Bytes blob with no trailing NUL. In-place ops:shift_frontmemmoves the tail to the head and drops n bytes (capacity preserved).clearsets len=0 (capacity preserved).snapshotcopies the current[0..len)into a fresh Bytes blob in the bus payload arena, builder unchanged.viewreturns a non-owning Bytes pointer aliasing the builder’s inline[i64 len][u8 data]region — zero allocation, zero copy; lifetime valid until the next mutation on the source builder.freedisposes the malloc-backed buffer.F.30 type promotion. The Hale-visible method surface returns
BytesView/StringView(typecheck-distinct fromBytes/String). The view-to- owned upgrade paths (std::bytes::clone,std::str::clone) are backed bylotus_bytes_clone(arena, src)(new) andlotus_str_clone(arena, src)(existing m49).F.30b view layout + epoch guard (2026-05-22 PM compaction). The
_view/_text_viewC primitives return a 16-byte by-value struct — no arena allocation in the hot path. Pre- compaction was a 24-byte struct heap-allocated per call, and that allocation was the dominant residual chunk- allocation trigger in long-running recv loops:#define LOTUS_VIEW_EPOCH_STATIC ((int64_t)-1)typedef struct lotus_view {void *src; // builder ptr (epoch >= 0, real view)// OR static data ptr (epoch == -1,// static-lifetime view from// lotus_view_from_static_data or// the null-handle path of// builder_view / builder_text_view).int64_t epoch; // stamped mutation_epoch, or the// static sentinel.} lotus_view_t;The
{void*, int64_t}layout fits SysV AMD64’s “two INTEGER eightbytes ≤ 16 bytes” return-by-value rule — both registers (rax,rdx) carry the view; arg-by-value passes in two integer arg registers. The underlying data pointer is recomputed at unpack time from((lotus_bytes_builder_t*)v.src)->buf(Bytes-shape:buf - 8; C-string shape:buf), so the view itself doesn’t need to store it.lotus_bytes_builder_tgains anint64_t mutation_epochfield bumped by every mutating op (append,append_slice,shift_front,clear,advance). Codegen at view-coerce sites emits a call tolotus_bytes_view_data/lotus_str_view_data, which compares the stamped epoch against the live epoch and callslotus_view_stale_panic(noreturn — stderr +_exit(1)) on mismatch. The 5b literal-default coercion callslotus_view_from_static_datato construct the view in-register with the static sentinel; the helpers skip the epoch check on that branch and returnv.srcdirectly as the underlying data pointer.Memory layout (Phase-2 (1)). The builder header is
{cap, buf, mutation_epoch}; the data area is preceded inline by an 8-byte length prefix matching the Bytes ABI:malloc'd region: [int64_t len][u8 data[cap]][NUL]^bufview(b)returns a 16-bytelotus_view_twhosesrcfield is the builder pointer; the read-site helper recomputes the data pointer asb->buf - 8(Bytes-shape, suitable forlotus_bytes_len/lotus_bytes_at/lotus_bytes_data). Append / append_slice / shift_front / clear / advance all update the inline prefix in sync with the data mutation AND bumpmutation_epoch. Cost: one extra pointer dereference per len access vs the prior{cap, len, buf*}shape, plus a one-load epoch check at every view-coerce site. Zero arena allocation per view() call (the dominant residual chunk-alloc trigger pre- compaction).lotus_str_builder_t(forstd::str::*) keeps the prior layout — no view surface there yet.These primitives are no longer the user-facing surface; they’re the C externs called by the
std::bytes::BytesBuilderstdlib locus (crates/hale-codegen/runtime/stdlib/bytes_builder.hl). Seespec/decisions.md§ F.28 for the rationale and the locus’s method shape. The locus-side calls reach these via internalstd::bytes::builder::__*path-call dispatch.ABI notes.
_newtakesint64_t initial_cap(previously zero-arg, hardcoded 64) — values<= 0are treated as the legacy default._appendreturnsint64_t status(1=ok, 0=fail on realloc-NULL or null-handle) — previously void; the status return is what the locus’sappendmethod checks before routing throughviolate alloc_failedper F.27.Builder handles are NOT layout-compatible with regular Bytes blobs. The struct shape is
{ size_t cap; size_t len; char *buf; }(24 bytes); Bytes blobs are[int64_t len][u8 data[]]. Solotus_bytes_at(builder, i)/lotus_bytes_len(builder)read the wrong slots if a builder handle is passed as a Bytes value. The Hale-level enforcement (BytesBuilderas its own locus type) makes that mistake impossible to express; this note is the C-side mirror — anyone calling these primitives directly from C / Rust must keep the distinction. -
lotus_tcp_recv_into(fd, builder, max_bytes) -> i64/lotus_tls_recv_into(handle, builder, max_bytes) -> i64/lotus_udp_recv_into(fd, builder, max_bytes) -> i64— 2026-05-19 (Phase 1, pond/websocket follow-up). Caller-provided destination at the syscall layer. Reads directly into the builder’s tail; grows on insufficient headroom; bumps the builder’s len by the count read. Return semantics mirror POSIX read(2):> 0bytes appended,= 0peer closed cleanly (TCP) / zero-length datagram (UDP),< 0error. EINTR retried internally. ASO_RCVTIMEOtimeout is distinguished from a fatal error:-2= “would-block / timed out, retryable”;-1= fatal (TCP:EAGAIN/EWOULDBLOCK; TLS:SSL_read→SSL_ERROR_WANT_READ/WANT_WRITE). The-2only arises when the caller has set a recv timeout (opt-in viaset_recv_timeout), so it’s backward-compatible — a caller that treats all< 0as error keeps working; a liveness loop checks for-2to run its ping/pong instead of tearing the connection down. No allocation ing_bus_payload_arena. No allocation ing_bus_payload_arena— closes the residual ~80% of the pond/websocket recv-loop leak that Phase 0’s in-place builder ops surfaced (the syscall layer’s own[i64 len][body]blob per call). Helperslotus_bytes_builder_reserve(handle, n)+lotus_bytes_builder_advance(handle, n)factor the grow + offset-bump solotus_tls.c(separate translation unit) can implement its recv_into without seeing the builder struct layout. -
lotus_str_lower(s) -> char*/lotus_str_upper(s) -> char*— ASCII case folding. One-pass byte-level fold; non-ASCII bytes pass through unchanged. Allocates in the bus payload arena. Used byhttp_request_headerfor RFC 7230 case-insensitive lookup. -
lotus_str_trim(s) -> char*— strip ASCII whitespace (space / tab / CR / LF) from both ends. Arena-anchored. -
lotus_str_replace(s, needle, rep) -> char*— greedy non-overlapping substring replace. Two-pass (count, then fill) so the output is right-sized in one arena alloc. Empty needle is a no-op. -
lotus_str_repeat(s, n) -> char*— n copies of s concatenated; n <= 0 returns empty. Single arena alloc. -
lotus_str_pad_left(s, width, pad) -> char*/lotus_str_pad_right(s, width, pad) -> char*— width-aligned output using the first byte ofpad(default space) as the fill char. No truncation:len(s) >= widthreturns s unchanged.
Process control
Section titled “Process control”- Exit codes.
main()returning()exits 0; returningintexits with that code. Panics exit non-zero. - Signal handling. SIGINT / SIGTERM trigger
drain→dissolveon the root locus. Stdlib provides finer-grained control if needed. - SIGPIPE globally ignored (added 2026-05-17, C2). The
prelude installs
signal(SIGPIPE, SIG_IGN)once atlotus_io_initso writes to a closed pipe (subprocess stdin, closed TCP socket, etc.) surface asEPIPEthrough the IoError channel instead of synchronously killing the parent. Applies process-wide — no opt-out. - Subprocess lifecycle (added 2026-05-17, C2 — see
spec/stdlib.md§ std::process for the API surface). Every spawned child gets its own process group viasetpgid(0, 0)in the post-fork prelude. Chosen overprctl(PR_SET_PDEATHSIG, SIGKILL)for POSIX portability (macOS / BSD parity); a controlledChild.dissolve()covers the orderly-shutdown path.Child.dissolve()closes the three pipe fds and kill-escalates idempotently (SIGTERM → 100 ms grace → SIGKILL → waitpid;ESRCH/ECHILDcount as success) so an unwaited child doesn’t leak zombies on scope exit. Thestd::process::runsynchronous form drains stdout + stderr via interleavedpoll()so the child can write to either stream without deadlocking; 16 MiB cap per stream.
stdout buffering
Section titled “stdout buffering”stdout is line-buffered for the lifetime of the program,
regardless of whether it’s attached to a TTY or a pipe. The
main prelude calls setvbuf(stdout, NULL, _IOLBF, 0) once
before any user code runs.
The default libc behavior (fully-buffered when stdout isn’t a
TTY) silently dropped output for any program that printed then
blocked on a syscall — println("READY"); accept_loop(); made
“READY\n” invisible to a piped consumer until the buffer
filled or the program exited. Test oracles, supervisors waiting
for a READY handshake, and log tailers all hung. Line-buffering
matches Python python -u discipline and Go’s default; \n-
terminated println calls flush immediately under any stdout
target.
stderr is line-buffered by POSIX already; the runtime doesn’t touch it.
What’s NOT in the runtime (lives in stdlib instead)
Section titled “What’s NOT in the runtime (lives in stdlib instead)”- Specific bus transports (NATS, UDP, etc.)
- File I/O
- Networking (sockets, HTTP)
- JSON / protobuf / msgpack encoding
- Most collections beyond what the language has built-in
- Math beyond
sum/prod(which are language-native) - Statistics
- Linear algebra
- String manipulation beyond literal handling
- Time arithmetic beyond comparison and arithmetic
- Logging / metrics / tracing
These are bundled with the toolchain (no separate install) but
require explicit import std::....
Form-vec runtime (v1.x-FORM-1)
Section titled “Form-vec runtime (v1.x-FORM-1)”The @form(vec) form lowers to a contiguous growable buffer
implemented in C. See spec/forms.md for the form contract and
synthesized method set; this section documents the runtime
shape.
C struct layout
Section titled “C struct layout”Each @form(vec) locus’s heap slot lowers to an inline
struct:
typedef struct { size_t cap; // allocated capacity (elements) size_t len; // number of valid elements char *buf; // contiguous element array} lotus_vec_<T>_t;The <T> suffix is conceptual — codegen monomorphizes per
cell type T, but the runtime primitives operate on the
common prefix layout via void * casts. All lotus_vec_*_t
typedefs share the {cap, len, buf} prefix.
Primitive functions
Section titled “Primitive functions”Defined in crates/hale-codegen/runtime/lotus_arena.c
(v1.x-FORM-1 PR4):
| Function | Behavior |
|---|---|
void lotus_vec_init(void *v) |
Zero-init: cap=0, len=0, buf=NULL |
void lotus_vec_push(void *v, size_t es, const void *x) |
Append; doubles cap on overflow |
int lotus_vec_get(void *v, size_t es, int64_t i, void *out) |
Bounds-checked read; returns 1=OK, 0=out-of-bounds |
int lotus_vec_set(void *v, size_t es, int64_t i, const void *x) |
Bounds-checked in-place write; returns 1=OK, 0=out-of-bounds (does not extend the vec) |
int lotus_vec_pop(void *v, size_t es, void *out) |
Returns 1=OK, 0=empty |
int64_t lotus_vec_len(void *v) |
Element count |
int lotus_vec_is_empty(void *v) |
1=empty, 0=non-empty |
void lotus_vec_destroy(void *v) |
free(buf); called at locus dissolve |
void lotus_vec_sort_int(void *v) |
In-place ascending sort of an int64_t-cell vec via qsort |
void lotus_vec_sort_float(void *v) |
In-place ascending sort of a double-cell vec; NaN treated as equal-to-anything |
void lotus_vec_sort_string(void *v) |
In-place ascending sort of a char *-cell vec under strcmp ordering |
void lotus_vec_sort_by(void *v, size_t es, int (*cmp)(const void *, const void *, void *), void *cookie) |
qsort_r wrapper; cmp is a codegen-synthesized per-(cell_type, direction) trampoline |
es (elem_size) is the cell type’s size in bytes — codegen
passes sizeof(T) at each call site.
Growth policy
Section titled “Growth policy”- Initial: cap=0, no allocation at locus birth.
- First push: allocates a 4-element buffer.
- Each overflow: doubles cap;
reallocs. Old contents copied by realloc; previous buf freed. - Shrink: not implemented in v1. Buf released at dissolve.
Failure shapes
Section titled “Failure shapes”lotus_vec_get/lotus_vec_set/lotus_vec_popreturn 0 on contract break (out-of-bounds / empty). Codegen wraps this into theTy::Fallible { success: T, payload: IndexError }surface (Unit-success forset) via a small adapter that synthesizes theIndexErrorstruct from the bool + the call args (shipped v1.x-FORM-2 PR5/6;setadded 2026-05-16).lotus_vec_pushOOM routes through the substrate-trap → closure-violation channel per the two-channel rule (shipped v1.x-FORM-2 PR6).- Sort family (
sort,sort_by,sort_desc_by, added 2026-05-16) is infallible from the language surface;sort_*C wrappers do not return a status code. If the user-supplied comparator insort_byfaults (a fallible call inside the comparator body raised throughor raise), the fault propagates andqsort_rstops mid-sort — the vec is left with every element still present, ordering partially applied.
Form-hashmap runtime (v1.x-FORM-4)
Section titled “Form-hashmap runtime (v1.x-FORM-4)”The @form(hashmap) form lowers to an intrusive open-addressing
hash table implemented in C. See spec/forms.md for the form
contract and synthesized method set; this section documents the
runtime shape.
C struct layout
Section titled “C struct layout”Each @form(hashmap) locus’s pool slot lowers to an inline
struct:
typedef struct { size_t cap; // power-of-two slot count size_t len; // live entry count size_t key_size; // sizeof(K), set at init size_t value_size; // sizeof(S), set at init int key_type_tag; // 0 = Int, 1 = String char *slots; // cap * (1 + key_size + value_size) bytes} lotus_hashmap_t;Each slot is 1 + key_size + value_size bytes:
[occupied: u8] [key: key_size bytes] [value: value_size bytes]occupied = 0 means empty. Backward-shift deletion (no
tombstones) — probes terminate at the first empty slot.
The C ABI is type-erased: codegen passes key_size /
value_size at init time, and per-call sites pass raw void *
key/value pointers. Codegen GEPs the indexed-by field on the
caller’s side to derive the key pointer before each set.
Primitive functions
Section titled “Primitive functions”Defined in crates/hale-codegen/runtime/lotus_arena.c
(v1.x-FORM-4 PR4):
| Function | Behavior |
|---|---|
void lotus_hashmap_init(void *m, size_t key_size, size_t value_size, int key_type_tag) |
Allocate cap=8 slots, zero them; freeze key/value sizes and key-type tag |
void lotus_hashmap_set(void *m, const void *key, const void *value) |
Insert or replace; grow at load factor 0.7 |
int lotus_hashmap_get(void *m, const void *key, void *out_value) |
Bounds-checked read; returns 1=OK, 0=missing_key |
int lotus_hashmap_has(void *m, const void *key) |
1=present, 0=missing |
int lotus_hashmap_remove(void *m, const void *key) |
1=removed, 0=missing |
int64_t lotus_hashmap_len(void *m) |
Live entry count |
int lotus_hashmap_is_empty(void *m) |
1=empty, 0=non-empty |
void lotus_hashmap_destroy(void *m) |
free(slots); called at locus dissolve |
Key types and hashing
Section titled “Key types and hashing”key_type_tag |
Type | Hash function | Equality |
|---|---|---|---|
0 (LOTUS_HASHMAP_KEY_INT) |
int64_t |
Knuth multiplicative (k * 0x9E3779B97F4A7C15) |
== on i64 |
1 (LOTUS_HASHMAP_KEY_STRING) |
const char * (NUL-terminated) |
FNV-1a over the bytes | strcmp == 0, with pointer-identity fast path |
Other key types (Bytes, custom structs, enum tags) are not
supported at v1; codegen rejects @form(hashmap) with a
focused diagnostic when the indexed-by field’s resolved type
doesn’t map to one of these two tags.
Growth policy
Section titled “Growth policy”- Initial:
cap=8, slots calloc’d at locus birth vialotus_hashmap_init. - Growth: when
(len + 1) > 0.7 * cap, double cap and rehash every live entry through the normalsetpath (the probe sequence changes with the new mask, so we don’t copy raw bytes between tables). - Shrink: not implemented in v1.
- Cap is always a power of two so hash-to-index folds to
& mask.
Deletion policy
Section titled “Deletion policy”Backward-shift deletion (no tombstones). After clearing the
target slot, the runtime walks forward through the cluster and
shifts any entry whose natural position is “before” the freed
slot in the probe sequence. The cluster boundary is the first
empty slot encountered. This keeps probe chains tight and lets
find_slot terminate correctly without a separate tombstone
marker.
Failure shapes
Section titled “Failure shapes”lotus_hashmap_get/lotus_hashmap_removereturn 0 on contract break (missing key). Codegen wraps this into theTy::Fallible { success: S, payload: KeyError }surface via the same machinery@form(vec)uses forIndexError, synthesizing theKeyError { kind: "missing_key" }payload at the call site (shipped v1.x-FORM-4 PR5).lotus_hashmap_setOOM during the slot calloc / realloc routes through the substrate-trap → closure-violation channel per the two-channel rule.
Form-ring-buffer runtime (v1.x-FORM-5)
Section titled “Form-ring-buffer runtime (v1.x-FORM-5)”@form(ring_buffer, cap = N) lowers a pool capacity slot to an
inline lotus_ring_buffer_t and synthesizes a fixed-capacity
FIFO surface (push / pop / len / is_full). The cap is baked in
at lotus_ring_buffer_init from the form annotation arg; the
backing buffer is malloc’d once at locus birth and never grows.
C struct layout
Section titled “C struct layout”typedef struct { size_t cap; // fixed at init; never changes size_t head; // index of oldest element (next pop) size_t len; // current element count, 0..=cap size_t elem_size; // bytes per element char *buf; // cap * elem_size bytes} lotus_ring_buffer_t;Codegen emits the matching LLVM inline struct on the locus’s
pool slot; the slot’s struct field IS the ring buffer (no
indirection). Element-size is sizeof(T) from the cell type’s
LLVM size_of.
Primitive functions
Section titled “Primitive functions”Defined in crates/hale-codegen/runtime/lotus_arena.c
(v1.x-FORM-5):
| Function | Behavior |
|---|---|
void lotus_ring_buffer_init(void *rb, size_t cap, size_t elem_size) |
malloc(cap * elem_size); head=len=0 |
int lotus_ring_buffer_push(void *rb, const void *src) |
1=pushed, 0=full; wraps modulo cap |
int lotus_ring_buffer_pop(void *rb, void *out) |
1=popped, 0=empty; advances head |
int64_t lotus_ring_buffer_len(void *rb) |
Current element count |
int lotus_ring_buffer_is_full(void *rb) |
1=full (len==cap), 0=not |
void lotus_ring_buffer_destroy(void *rb) |
free(buf) at locus dissolve |
Failure shapes
Section titled “Failure shapes”pushreturns 0 when full → codegen converts to Bool false at the language surface (fn push(x: T) -> Bool).popreturns 0 when empty → codegen lazily allocates anEmptyError { kind: "empty" }payload on the err path, surfaced to the caller’sorclause.- OOM during init (cap × elem_size too large to malloc) leaves
buf == NULL; subsequent push/pop see a 0-cap buffer and refuse / fail. Routing OOM through the closure-violation channel is deferred to a future hardening pass — the v1 contract is “fixed cap; if init can’t allocate, the buffer is permanently empty.”
Native codegen defaults
Section titled “Native codegen defaults”What the compiler emits for a native hale build:
- Host-CPU tuning + O3 by default. Native builds tune to the
host CPU (
target-cpu/target-featuresfrom the build machine) and run LLVM’s aggressive (O3) pipeline — both the module passes and the backend codegen level. This unlocks autovectorization across all generated code (e.g. AVX-512 on a capable host). Consequence: a native binary is not portable across microarchitectures — it may use instructions absent on an older CPU. --target-cpu native | baseline.native(default) is the host-tuned build above.baselinepins a portablex86-64-v3target (AVX2 + BMI2 + FMA) for distributed artifacts that must run on any modern x86-64 CPU. The emitted module self-describes its subtarget via per-functiontarget-cpu/target-featuresattributes, so the choice is carried into bitcode (it survives LTO).LOTUS_LTO=1— opt-in full-LTO. Read at build time. Emits the Hale module as LLVM bitcode and compiles the lotus C runtime TUs with-flto, so the finalclang -flto -O3 -fuse-ld=lldlink inlines the runtime hot paths (arena bump-allocator, string helpers, shm_ring framing) across the TU boundary into the Hale-generated callers — a boundary that’s otherwise opaque. Worth a few percent on allocation/coordination-heavy code; neutral on already-vectorized loops (the host tuning is preserved under LTO via the function attributes above). Off by default: the LTO link is ~3–4× slower and requireslldon PATH. Native, non-sanitizer builds only;wasm32and sanitizer builds keep the ordinary non-LTO link. The-Wl,--wrapmalloc/syscall shims (and theLOTUS_ARENA_LOG_BIG_CHUNKS/std::diag::syscall_countfeatures that ride them) are preserved under LTO —lldresolves--wrapbefore LTO codegen.wasm32is unaffected — it staysgeneric/O2 (the browser bundle is size/compat-sensitive).
Diagnostic + tuning env vars
Section titled “Diagnostic + tuning env vars”A small set of env vars toggle runtime instrumentation and glibc tuning hooks. All are opt-in; unset (the default) keeps the runtime quiet.
| Env var | Effect |
|---|---|
LOTUS_ARENA_LOG_BIG_CHUNKS=<N> |
Logs every arena chunk + libc allocator (malloc / realloc / calloc / mmap) >= <N> bytes to stderr with size, monotonic seqno, and 8-frame backtrace. Use 1 (= 1 MiB) as a shortcut; any positive decimal byte count works (e.g. 4096). Each event labeled by source: arena_big_chunk, malloc_big, realloc_big, calloc_big, mmap_big. Note: only fires on the fresh-malloc path; chunks recycled from the per-thread pool bypass this hook — use LOTUS_ARENA_LOG_CHUNK_ATTACH for the full picture. |
LOTUS_ARENA_LOG_CHUNK_ATTACH=<N> |
Logs every chunk attachment to ANY arena — both fresh-malloc (chunk_attach_malloc) AND per-thread-pool-recycled (chunk_attach_pool) paths — when cap >= N. Use 1 for “log every chunk attachment”. Each event additionally prints arena=<ptr> kind=<root|sub> label=<resolved>: root means the chunk attached to a top-level locus-lifetime arena (a leak class if it grows); sub means a subregion (method scratch / free-fn body) that will recycle to the pool on destroy. label walks the subregion→root chain and looks up the root in the residency registry — requires LOTUS_ARENA_RESIDENCY=1 to populate the label map. Filter kind=root label=<name> to isolate the actual arena-growers. Shares the LOTUS_ARENA_LOG_BIG_MAX_EVENTS cap with the big-chunks logger. |
LOTUS_ARENA_LOG_BIG_MAX_EVENTS=<N> |
Caps the log at <N> events per process. Default 200. Set to 0 for unlimited (useful when watching low-rate sub-MiB allocation patterns over a long window). |
LOTUS_CHUNK_POOL_STATS=1 |
Dumps per-thread chunk-pool hit / miss / store / overflow counters to stderr at process exit. Diagnostic for “pool isn’t recycling” symptoms — pairs hits vs misses, stores vs overflows. The atexit handler runs on the main thread; counters are __thread so the dump is that thread’s view. |
LOTUS_GLIBC_ARENA_MAX=<N> |
Calls mallopt(M_ARENA_MAX, <N>) at startup. Caps glibc’s per-thread malloc arena count. 1 forces a single arena (max contention, min virtual-address fragmentation); higher <N> trades contention for parallelism. Useful belt-and-suspenders against the per-thread arena heap-segment proliferation glibc default tuning can produce on long-running daemons. Unset keeps glibc’s default. |
LOTUS_BUS_PAYLOAD_ARENA_CAP=<N> |
Overrides the lazy-global bus payload arena’s byte cap (default 64 MiB). When the cap fires, lotus_arena_alloc returns NULL and the existing alloc-fail paths (empty_global / alloc_failed violation) surface degraded service rather than OOM-killing the process. |
LOTUS_ARENA_RESIDENCY=1 |
Registers every top-level arena (locus __arenas, g_bus_payload_arena, the program-wide global) into a side-table at creation time with a 24-frame construction backtrace. std::process::dump_arena_residency() walks the live set and emits one line per arena to stderr — bytes / chunks / parent / label, sorted by bytes desc — with the construction backtrace. Subregions (method scratch) are skipped; they destroy at method exit and don’t accumulate residency. Atexit also dumps, but post-dissolve fires after all loci tear down — useful only for the global arena’s final state. Long-running daemons should call dump_arena_residency from a heartbeat / checkpoint tick so locus arenas are sampled while still alive. |
LOTUS_CHUNK_POOL_PREFILL=<N> |
Per-thread chunk-pool pre-fill on first touch. Default 32 (= 2 MiB resident per scheduler thread). Set 0 to disable. Bumps the pool’s steady-state floor so brief bursts don’t drain to zero and miss into malloc; the trade-off is per-thread resident memory. |
LOTUS_TSAN=1 |
Read at build time (by the codegen’s build_executable, not at runtime). When set, the emitted clang command passes -fsanitize=thread for both the C runtime compile and the binary link, and skips the -Wl,--wrap=malloc/realloc/calloc/mmap shim surface (TSAN intercepts malloc itself; the wrap’d LOTUS_ARENA_LOG_BIG_CHUNKS diagnostic is silently no-op under TSAN). The resulting binary runs ~5-15× slower; use only for race-hunting workloads. The C runtime embeds an empty __tsan_default_suppressions hook at link time so no external suppression file is needed; all originally-flagged substrate races (bus queue drain, arena destroy, coop pool worker, env-var lazy-init) have been fixed and the suppression list is empty. Opt-in tests live behind #[ignore] and the env var (see crates/hale-codegen/tests/form_hashmap_lockfree_tsan.rs). |
LOTUS_LTO=1 |
Read at build time. Opt-in full-LTO native build: the Hale module is emitted as bitcode and the lotus runtime TUs compile with -flto, so the clang -flto -O3 -fuse-ld=lld link inlines the runtime hot paths (arena / string / shm_ring) across the TU boundary into the Hale callers. A few percent on allocation/coordination-heavy code, neutral on vectorized loops (host tuning preserved via per-function target-features). Off by default — ~3–4× slower link, requires lld. Native non-sanitizer only; --wrap shims survive (lld resolves them before LTO codegen). See Native codegen defaults above. |
LOTUS_BUS_LOG_UNMATCHED=1 |
Surfaces silent no-key-match drops in lotus_bus_local_dispatch_keyed (Phase 3 routing keys). When set, each publish that matches no where key == ... subscriber for the topic emits a single stderr line citing subject, key, and the per-topic subscriber counts (specific vs unkeyed). Off by default — the silent-drop is correct for on_unmatched: swallow topics in steady state, but during bring-up the lack of any signal is load-bearing on debug cycles. Implied by LOTUS_BUS_LOG_DROP=1. |
LOTUS_BUS_LOG_DESERIALIZE_DROP=1 |
Surfaces silent drops in the udp:// reader thread when (a) no deserializer is registered for the inbound subject, or (b) the deserializer returns <= 0 (size mismatch, bounded-read failure). Emits one stderr line per drop naming the subject, the payload size, and (when applicable) the deserializer’s return value. Off by default; the silent-skip on cross-routed multicast noise is the correct steady-state behavior. Same env-gated pattern as LOTUS_BUS_LOG_UNMATCHED for keyed-dispatch misses. Implied by LOTUS_BUS_LOG_DROP=1. |
LOTUS_BUS_QUEUE_CAP=<N> |
Caps the cooperative bus dispatch queue, each per-pinned-locus mailbox, and each cooperative pool’s queue at N cells (default 8192; floor 64; rounded up to a power of two; read once). v0.9.0 footprint change: the pinned mailbox and cooperative-pool queues are now lock-free MPSC rings (Vyukov bounded ring + signal-only-when-parked wake), and a fixed-size lock-free ring pre-allocates its cap up front rather than growing to it — so each pinned subscriber mailbox and each cooperative pool now costs ~4.3 MB resident at the default cap (vs the prior grow-as-needed). With the typical handful of pinned loci / pools this is a few-to-low-tens of MB; lower LOTUS_BUS_QUEUE_CAP for pinned-/pool-heavy programs to shrink it (the rings honor it identically). When a producer hits the cap it back-pressures instead of growing without bound (GH #125) — every message is still delivered. The mechanism: a single-threaded producer on the cooperative queue inline-drains it to free space; a cross-thread producer to a full ring blocks (a fenced producers-waiting handshake) until the single consumer drains a slot; a handler self-publishing to its own full ring spills to a consumer-thread-local overflow list (it can’t block on itself). The cross-cooperative-pool shared queue path (multiple drainers, no single consumer) is the remaining non-lock-free path — a follow-on. Lower the cap to tighten the bound / footprint; raise it to reduce drain bursts at the cost of resident memory. |
LOTUS_UNIX_STREAM=1 |
GH #231: forces the unix bus transport into framed SOCK_STREAM mode on Linux (the Darwin default — macOS has no AF_UNIX SOCK_SEQPACKET). Wire format per message: [u64 LE payload len][u64 LE seq][payload]; the seq is per-connection monotonic from 1, reset per accepted peer, and the receiver counts gaps (seq_gaps counter — GH #236’s loss-computability primitive). Set for EVERY process on a socket: framed and SEQPACKET ends don’t interoperate (a mismatch trips the 8 MB length sanity cap with a diagnostic naming the likely cause). Primary use: Linux CI/test coverage of the macOS code path. |
LOTUS_BUS_COUNTERS_DUMP=1 |
GH #236 (observability groundwork): prints one stderr line per remote binding at teardown — [bus counters] subject=... kind=... role=... sent= delivered= bytes_sent= bytes_delivered= send_failures= dropped_lost= waits= rearms= reconnects=. The counters are plain relaxed atomics bumped at the transport choke points (fanout send, serve-loop dispatch, re-arm, reconnect) and exist as the substrate for the iris observer; the dump is the operator/test surface until an in-process consumer ships. dropped_lost counts publishes made while a connect binding was in the lost/reconnecting window (GH #233 — drops the publish contract makes deliberate and visible). waits counts publishes that parked in or wait through that window instead (GH #255 phase 1). buffered_early / dropped_early (GH #468) count boot-window wire messages buffered before any matching subscriber registration existed, and the subset dropped (buffer eviction at the 64-msg/1 MiB per-binding cap, a deserialize failure at flush, or a subject that never gained a subscriber by teardown). Socket-buffer occupancy is intentionally not a counter: it’s a poll-time SIOCOUTQ query against the live fd. |
LOTUS_BUS_LOG_DROP=1 |
Broad superset for diagnosing “publish appears to succeed but handler doesn’t fire” symptoms. Implies LOTUS_BUS_LOG_UNMATCHED + LOTUS_BUS_LOG_DESERIALIZE_DROP AND covers additional silent-drop sites the narrower vars miss: lotus_bus_dispatch’s serialize-fn-returns-<=0 case, the local-fanout (lotus_bus_dispatch_wire + lotus_bus_local_dispatch) zero-matching-subscribers case, per-entry deserialize-returns-<=0 on the local-fanout path, and the no-post-target case (mailbox / coop_pool / global queue all NULL on a matched entry). Each line names the call site, subject, and relevant size / index info so a bus-heavy repro can identify exactly which silent-skip is firing. Reach for this first when investigating bus-drop friction; the narrower vars stay supported for their specific bring-up scenarios. |
LOTUS_BUS_QUIESCE_MS |
GH #468: bound on the main-exit ingress quiesce (default 500, 0 disables it). At every main-exit point, before pools join and loci dissolve, LISTEN binding fds half-close and their readers drain kernel-accepted data to true EOF through the still-intact registry; this bounds how long exit waits for a wedged reader. Exceeding the bound is loud (stderr names it) and drops whatever stayed undrained. Test-only siblings LOTUS_BUS_TEST_BOOT_HOLD_MS / LOTUS_BUS_TEST_READER_STALL_MS deterministically stretch the boot-registration window / the reader’s descheduled window (the loss canaries in binding_ingest_468.rs use them); never set them in production. |
Every top-level arena is created via lotus_arena_create_labeled(name) and carries an immutable human-readable label string. The codegen passes the locus name (e.g. WsClient, __lib_metrics_metrics_MetricMap); the program-wide global is labeled lotus.arena.global; g_bus_payload_arena labels itself. The label is the load-bearing identifier in the residency dump; backtraces resolve via -rdynamic for cases where the label alone isn’t enough.
The arena-chunk pool and -wrap=malloc family ship in every
binary unconditionally; the env vars are zero-cost when unset
(one int read + one branch per allocation). The -rdynamic
link flag is similarly unconditional so backtrace symbols
resolve without addr2line.
Runtime size budget
Section titled “Runtime size budget”The runtime should be small enough that a hello-world program binary is < 1 MB statically linked, and < 100 KB if dynamic linking against libc. This is a target, not a guarantee.
The framework’s discipline enables this: no GC, no metadata overhead per allocation, region-based MM compiles to bump allocators. Comparable to C in size, with ergonomics closer to Erlang.
Open questions for runtime
Section titled “Open questions for runtime”- Async / await integration. Reserved keywords, no v0 semantics. The lifecycle state machine + cooperative yield points subsume most of what async is for; explicit async/await may not be necessary.
- FFI to existing languages. Generic FFI in stdlib; team-specific bindings (e.g. domain-specific typed messages) live as third-party packages. Marshalling helpers in stdlib.
- Hot-reload of code (not just perspectives). Erlang supports module-level hot reload. Lotus’s perspective hot-reload is more granular and addresses most of the use case; full code hot-reload may not be needed.
- Determinism mode for tests. Resolved for the single-pool
case: no mode is needed, because single-pool execution is
deterministic by construction, and that is now a stated,
pinned guarantee rather than an accident — see
spec/testing.md§ Determinism. What remains open is deterministic re-execution of multi-pool programs, which is the record/replay track (GH #296) — reproducing a recorded per-consumer order, never constraining live scheduling.