Operations & debugging

Most of the time a Hale program either works or fails loudly. The two exceptions — the ones that send you here — are a message that quietly doesn’t arrive and resident memory that quietly grows. Both are silent by design (the steady-state behavior is correct), so the runtime ships opt-in diagnostics you switch on with an environment variable or a build flag. This chapter is the operator’s map: what each knob shows, and two worked triage walkthroughs.

Nothing here changes behavior — every switch is observe-only. The canonical reference for each variable is spec/runtime.md; this is the pedagogical version.

Bus: “my publish isn’t arriving”

A publish that compiles is not a publish that’s delivered — the subject might match no subscriber, the payload might fail to deserialize, or the subscriber might be on a pool that never runs. The bus drops these silently because for an on_unmatched: swallow topic in steady state that is the right behavior. To see the drops, set one variable:

LOTUS_BUS_LOG_DROP=1 ./myapp

LOTUS_BUS_LOG_DROP is the broad net — reach for it first. It prints one stderr line at every silent-drop site, naming the call site, subject, and size/index info: no-matching-subscriber, serialize-returned-≤0, deserialize-returned-≤0, and the “matched-but-no-post-target” case (mailbox / pool / queue all null). It implies the two narrower variables, which you can use on their own once you know which class you’re chasing:

Variable	Surfaces
LOTUS_BUS_LOG_DROP=1	everything below, plus serialize-fail and no-post-target
LOTUS_BUS_LOG_UNMATCHED=1	a keyed publish (where key == …) that matched no subscriber — prints subject, key, and the per-topic subscriber counts
LOTUS_BUS_LOG_DESERIALIZE_DROP=1	the udp:// reader thread dropping a frame (no deserializer registered, or a size-mismatched read)
LOTUS_BUS_COUNTERS_DUMP=1	one line per remote binding at exit: messages/bytes sent and delivered, send failures, publishes dropped while a link was down, publishes that parked on or wait, listener re-arms, reconnects

The shape that produces no line at all. If LOTUS_BUS_LOG_DROP is silent but the handler still never fires, the message was delivered to the queue and the problem is downstream: the subscriber’s pool isn’t draining. The classic cause is a run() on a cooperative pool that blocks (a long time::sleep, a blocking syscall) and starves the handler — hale check warns on blocking syscalls in a cooperative run(), and std::process::dump_pool_residency() shows pending counts per pool so you can see work piling up unserved.

Memory: “my RSS is growing”

Hale frees a locus’s whole region on dissolve, so a leak is usually one of two things: an allocation that escapes to a long-lived arena (it never dissolves), or a queue/buffer whose high-water mark keeps climbing. Two layers of instrumentation pin it down — one at runtime, one at compile time.

Runtime residency. Set LOTUS_ARENA_RESIDENCY=1 to register every top-level arena (each locus’s region, the global, the bus payload arena) with a construction backtrace. Then call std::process::dump_arena_residency() to emit one line per live arena — bytes, chunks, parent, label, and used=, the bytes its chunks actually hold — sorted by bytes descending, each with the backtrace of where it was created. bytes= moves a whole chunk at a time; used= moves by the byte, so two dumps a known number of requests apart say what each request leaves behind:

// In a long-running daemon, sample from a heartbeat tick so locus
// arenas are caught *while alive* — the atexit dump fires only
// after every locus has torn down.
fn on_tick() {
    std::process::dump_arena_residency();   // → stderr, needs LOTUS_ARENA_RESIDENCY=1
    println("rss=", std::process::rss_bytes() / 1048576, " MB");
}

std::process::rss_bytes() is the cheap top-line number — poll it to confirm growth before you go digging. dump_pool_residency() is the per-pool view (pending/in-flight work), useful when the growth is a queue rather than an arena.

One caveat on rss_bytes(): it is getrusage’s peak, and a process inherits its parent’s peak through fork + exec — a worker your supervisor spawned reports at least the supervisor’s RSS from its first instruction. Poll the delta over the life of the process rather than the absolute number, or read /proc/self/statm field two (current RSS, no inherited history) when the figure has to be this program’s own. See spec/stdlib.md § std::process::rss_bytes().

Compile-time proofs. Before the program even runs, three build flags report on allocation shape:

Flag	Reports
(default on every check/build)	flag an allocation that escapes into an unbounded context and accumulates until its locus dissolves (advisory warnings; --no-warn-unbounded-alloc opts out)
--dump-alloc-summary	every allocation site, escape-tagged (local / returned / stored-to-self / sent), with the bounded-vs-unbounded verdict; plus each locus’s storage shape (capacity slots, @form, projection cap) and the self.<field> / self.<slot> an allocation targets
--dump-resource-budget	a static count of the threads placement spawns (pinned fields per replica and per live construction of main, plus adapters), worker pools, bus subjects and fd-acquisition sites, with what it does not count named (--check-resource-budget <file> gates it against declared ceilings)
--locality-report	per-locus working-set size against cache-tier budgets

The memory-bound warnings run by default on every hale check and hale build. Run-to-exit programs are exempt automatically: a binary whose main starts no run loop and subscribes no handler owes no memory-bound proof, so scripts and one-shot tools stay silent.

For a long-lived service, the surface is:

• @unbounded fn — the greppable in-source carve-out for an acknowledged accumulation (an operator-sized cache, an idempotency log). Silences that body’s sites. Also valid on a lifecycle hook (@unbounded run { … }).

  type Snapshot { seq: Int; total: Int; }
  
  locus Aggregator {
      // ... handlers checked for unbounded accumulation ...
  
      @unbounded fn on_snapshot(s: Snapshot) {
          // acknowledged: this cache is operator-sized on purpose.
      }
  }

• @bounded_by(<name>) — the same carve-out, for growth that is understood: it names the input that bounds it (@bounded_by(record) on an in-memory mirror of the record, @bounded_by(open_requests) on a queue of them). @unbounded means “not yet understood”; a fn carries one or the other.

• --no-warn-unbounded-alloc — opts a whole run out.

• --report-unbounded-use — prints one line per @unbounded or @bounded_by fn or hook, with how many survey sites and hot-path warnings it silences in this program, and the bound; the stdlib’s own decorators the program reaches are listed too, at crates/hale-stdlib/hl/<file>:<line>. A decorator whose counts are 0 in every program that compiles it acknowledges nothing and can go.

• @bounded locus L { … } is now redundant with the default and still accepted.

The warnings are advisory — they print but don’t fail the build. A warning here is the compile-time complement to the residency dump: it tells you which site can grow before you’ve watched it grow.

Bus backpressure: bounding a flood

A producer that outruns its consumer used to grow the dispatch queue without limit. It no longer does — the queue, each pinned-locus mailbox and each cooperative pool’s queue are capped at LOTUS_BUS_QUEUE_CAP cells (default 8192; a pinned mailbox or pool queue pre-allocates its cap, about 4.3 MB at the default):

LOTUS_BUS_QUEUE_CAP=1024 ./myapp   # tighter bound, more frequent drains

Past the cap the producer back-pressures rather than buffering: a single-threaded cooperative producer inline-drains the queue (runs the oldest handlers) to make space; a cross-thread producer to a pinned mailbox blocks on a condvar until the consumer drains a slot. Every message is still delivered — only the timing and memory profile change. Lower the cap to tighten the memory bound; raise it to reduce drain bursts.

Shelling out to other programs

Ops glue often means running another tool. std::process::run does a synchronous fork + exec + wait and captures the result. The argument vector is newline-separated (no shell, no word splitting — each line is one argv entry):

let out = std::process::run("git\nstatus\n--short") or raise;
println("exit ", to_string(out.code));
println(out.stdout);
if len(out.stderr) > 0 { println("stderr: ", out.stderr); }

The returned ProcessOutput carries code: Int (the exit code, or -1 if killed by a signal), signal: Int (the killing signal, 0 if it exited normally), and stdout / stderr as captured Strings. run is fallible(IoError) — a missing binary or a fork failure raises rather than returning a bogus output.

For a long-running child you drive incrementally, the lower-level spawn / wait / kill / write_stdin / read_stdout / read_stderr surface over a Child handle is in spec/stdlib.md.

Keeping the child: adopt

A supervisor has to hold its subprocess past the method that started it, and self.child = std::process::spawn(argv) is refused — a locus-typed field takes only a locus literal, because a store would leave the field and the frame that produced the value both claiming the instance. std::process::adopt is the transfer: it moves the spawned handle’s pid and pipe fds into the Child your field already owns, so there is only ever one owner.

locus Job {
    params { child: std::process::Child = std::process::Child { }; }

    fn start(argv: String) {
        let spawned = std::process::spawn(argv) or std::process::Child { };
        std::process::adopt(self.child, spawned);
    }

    fn tick() {
        let code = std::process::try_wait(self.child) or -2;
        if code != -2 {
            self.start("./worker");
        }
    }
}

Three things that follow from “one owner”, and that you get without writing any of it:

• adopt terminates what the field was holding (TERM → grace → KILL → reap), so restarting is just start again. Call wait/try_wait first if you need the outgoing child’s exit code.
• The replacement’s pipes are never closed by the old handle. The field’s descriptors stay open until adopt replaces them, so the kernel cannot have handed their numbers to the new child — the hazard that makes a hand-rolled version of this so easy to get wrong.
• A failed spawn leaves the field empty, not half-built: the or disposition hands back an empty Child { }, and adopting it clears the field. try_wait on an empty field answers “exited, code 0”.

Never copy pid / stdin_fd / … across by hand. Two Childs holding one descriptor triple both close it.

Teardown rides on the owner: when the locus holding the field dissolves, the field’s dissolve closes the pipes and reaps the child. A handle you only let-bind has no such owner — a factory-returned locus is never reclaimed, so its dissolve never runs and the child is still alive when your program exits. Moving it into a field is what gives it a teardown.

A supervising daemon reaps without blocking via std::process::try_wait(c) — -2 means still running (poll again on your next tick), any other value is the exit code (-1 = killed by a signal), and the child is reaped:

locus Supervisor {
    params { child: std::process::Child = std::process::Child { }; }

    fn tick() {
        let code = std::process::try_wait(self.child) or -2;
        if code != -2 { self.on_child_exit(code); }
    }

    fn on_child_exit(code: Int) {
        println("child exited: ", code);
    }
}

std::process::signal(c, sig) sends an arbitrary POSIX signal (15 = TERM, 1 = HUP for a config reload, …) when the fixed TERM→KILL escalation of kill is more than you want.

Other process self-introspection: std::process::pid(), std::process::uid() (the real uid; a unix::Rpc peer is named uid:<n>, so it is what a program compares a caller’s name against), std::process::exit(code), and std::process::rss_bytes() (peak RSS — see Memory above).

Worked triage

“My subscriber’s handler never runs.”

1. LOTUS_BUS_LOG_DROP=1 ./app. A line at the publish? → the subject or key doesn’t match, or the payload won’t deserialize. Fix the subject/key or the payload type.
2. No line, but still no delivery? → the message reached the queue; the consumer isn’t draining. Check the subscriber’s pool: a cooperative run() that blocks starves handlers. hale check flags blocking syscalls; dump_pool_residency() shows the pending pileup.
3. Subscriber is an inline child or on where async_io? → confirm it’s instantiated as an owned param or top-level, not unowned in a method body (which dissolves at scope exit before it can fire — hale check errors on this).

“My RSS climbs over hours.”

1. rss_bytes() from a heartbeat — confirm it’s monotonic, not sawtooth (sawtooth is healthy churn).
2. LOTUS_ARENA_RESIDENCY=1 + dump_arena_residency() from the same heartbeat — find the arena whose bytes grows. The label and backtrace name the locus and birth site.
3. A root-kind arena growing is the leak; a sub arena recycles. If it’s the bus payload arena, the high-water is queue depth — lower LOTUS_BUS_QUEUE_CAP. If it’s a locus arena, you’re accumulating into a field: prefer in-place mutation (self.f.x = v) over whole-value replace (self.f = T{…}), which bump-allocates fresh each time. --dump-alloc-summary names the site at compile time.

Observing a running system

Set LOTUS_OBS=1 and any hale binary publishes an observation segment — a shared-memory ring of records emitted from the runtime’s own choke points: bus publishes and deliveries (each attributed to the publishing/subscribing locus), transport sends and deliveries (paired across processes by a (sender-origin, sequence) key so a message’s send and its deliveries line up into a cross-process edge), and locus lifecycle (birth with parentage, dissolve, restart). It is dormant by default: with the env var unset every probe is a single predictable branch, and even enabled it writes only counters until an observer attaches (observer_count on the segment’s control page). One SPSC ring per emitting thread; a late-attaching observer gets the live locus tree replayed as births so it can reconstruct the supervision graph.

The segment lives at /hale-obs-<pid> with a registration file under $XDG_RUNTIME_DIR/hale/ (or /tmp/hale-obs/). The wire layout is the iris observation protocol — the canonical contract is spec/runtime.md § Native observation emission; iris is the reference consumer and ships in the binary — hale run --observe prog.hl runs the program with the segment published and an observer beside it (see Iris): one whose lifetime is bounded by that hale, and whose own output goes to stderr rather than into the program’s stdout. Knobs: LOTUS_OBS_RINGS (default 8, or 64 when recording), LOTUS_OBS_SLOTS (default 4096).

Cross-process edges opt into the wire. The (origin, seq) key that pairs a send with its deliveries travels in the wire message (a self-describing header on udp://; the frame header on unix:// under LOTUS_UNIX_STREAM=1). That is a wire-format change a stale peer running an older binary cannot parse, so LOTUS_OBS=1 alone never touches the wire — it gives you counters and single-process records with the wire byte-for-byte unchanged. To get cross-process edges, set LOTUS_OBS_WIRE=1 across the whole fleet; every node must be on a build that understands the header. (A non-framed unicast transport carries no seq and falls back to a local delivery count even with the wire enabled.)

Recording a run. LOTUS_OBS_RECORD=<path> turns the sampler into a flight recorder: every observation record is drained to <path>, and the disposition flips from “drop rather than stall” to “stall rather than drop” — a full ring blocks its producer until the drain catches up, and a run that couldn’t record everything fails loudly instead of producing a silently incomplete file. It implies LOTUS_OBS=1 and needs no observer attached. The recording captures each consumer’s actual handler order, every queued publish’s payload, and a journal of the nondeterministic reads (std::time, std::rand, std::os::getrandom, std::env). Recording changes your program’s timing by design; don’t leave it on in production. The file format is pre-stable (GH #296).

Replaying one.

hale replay run.halerec app.hl            # re-execute it
hale replay run.halerec app.hl --diff     # + compare, fail on any divergence
hale replay run.halerec app.hl --diff --json      # ...that verdict, machine-readable
hale replay run.halerec app.hl --at 65:12 # SIGSTOP at consumer 65's 12th consume
hale replay run.halerec app.hl --allow-truncated  # crashed run → replay the prefix
hale replay run.halerec app.hl --feed     # inject the ingress tape into changed code
hale replay run.halerec app.hl --dev      # the build options the recording was made under

replay recompiles the program, so it takes hale build’s options — as hale run does — and needs the ones the recording was made under: they are part of the execution identity, so a recording from hale run --dev is refused by a default hale replay and admitted by hale replay --dev, and one made under --env prod needs hale replay --env prod. A binary from hale build app.hl, run with LOTUS_OBS_RECORD set, records a run that hale replay ... app.hl admits: a build and hale run give one program one identity. So does a directory build: the recording of myapp/myapp replays under hale replay ... myapp/main.hl.

The full story — admission by executable identity, the safe-by-default effect gate (--allow-live-effects), env-value redaction (LOTUS_OBS_RECORD_ENV), the hermetic wire and ingress injection, feed mode (backtesting), crash-truncated recordings, and what the comparator actually compares (a match names its coverage per category, and says which categories a recording never exercised) — has its own chapter: Record & replay.

Debugging with the native toolchain

Hale binaries carry full DWARF by default (zero runtime cost): line tables and variable info. That means real debugging — stop AND inspect:

hale build myservice
gdb ./myservice
(gdb) break myservice.hl:42
(gdb) run
(gdb) backtrace          # real .hl file:line frames, inline stacks
(gdb) info args          # typed parameters: n = 21
(gdb) info locals        # typed lets: doubled = 42, frac = 0.5
(gdb) print msg          # Strings print their text: "hello!"

Hale scalars map to proper DWARF base types (Int, Float, Bool, Decimal, Time, Duration), String is a char* so debuggers print the contents, and struct-typed values carry full member info — p *r prints {key = "alpha!", n = 41, f = 2.5} with nested structs as typed pointers. A variable can read <optimized out> after its last use — that’s the optimizer, not missing debug info; hale build --dev keeps more of the frame live.

addr2line -e ./myservice 0x4a2f10 resolves crash-dump addresses to source lines, and ASAN reports carry file:line through both the Hale code and the runtime. One thing to know when chasing a suspected use-after-free: the runtime recycles a destroyed arena’s 64 KiB chunks through a thread-local pool, so a read from memory that was already reclaimed lands on bytes the process still owns and the sanitizer stays quiet — a wrong answer, not a report. Run with LOTUS_NO_CHUNK_POOL=1 to hand every chunk straight back to libc; the same read then aborts with a heap-use-after-free naming where it was freed and where it was read. An AddressSanitizer build has this on by default. Profile with perf record --call-graph dwarf (frame pointers are deliberately not forced — they cost ~22% on runtime fast paths). Opt out of debug info with LOTUS_NO_DEBUGINFO=1.