Observing a program

Hale programs expose four observability surfaces: structured application and runtime logs, Prometheus metrics (std::observe::Scrape), HTTP health and…

Hale programs expose four observability surfaces: structured application and runtime logs, Prometheus metrics (std::observe::Scrape), HTTP health and queue-capacity probes (std::observe::Probes), and OpenTelemetry lifecycle events (std::observe::Otlp). The three network backends are declared as params on your main locus and incur zero binary or runtime cost when omitted.

main locus App {
params {
probe_port: Int = 9090;
reg: std::metrics::Registry = std::metrics::Registry { namespace: "app" };
metrics: std::observe::Scrape = std::observe::Scrape { port: 9100 };
probes: std::observe::Probes = std::observe::Probes { port: self.probe_port };
events: std::observe::Otlp = std::observe::Otlp { };
}
placement { metrics: pinned; events: pinned; }
run() {
let hits = std::metrics::counter(self.reg, "hits",
std::metrics::labels_empty());
hits.inc();
while !self.draining { std::time::sleep(100ms); }
}
}
fn main() { App { }; }

Observability Surfaces Summary

Section titled “Observability Surfaces Summary”
Surface / Backend Activation Endpoints / Output Purpose
Application & Runtime Logs std::log sinks + automatic runtime stderr stdout / stderr / FileSink (HALE_LOG, HALE_LOG_FORMAT) Structured logfmt, json, or text records for app events and lotus.* runtime diagnostics
Local Shared Memory LOTUS_OBS=1 or hale run --observe Shared-memory segment read by Iris Live inspection of the locus tree and bus traffic with zero code changes
std::observe::Scrape main locus param GET /metrics (default 127.0.0.1:<port>) Prometheus text exposition of all std::metrics::Registry instances and hale_* runtime metrics
std::observe::Probes main locus param GET /healthz, /readyz, /capacity (0.0.0.0:<port>) Liveness (queue stall detection), readiness, and queue utilization JSON
std::observe::Otlp main locus param POST <endpoint> (OTLP/HTTP JSON logs) Exports hale.start, locus.birth, locus.dissolve, locus.restart, and locus.failure

Default Behavior (With Nothing Declared)

Section titled “Default Behavior (With Nothing Declared)”

When a program declares no std::observe::* parameters on its main locus, no network ports are opened and no telemetry is sent over the network:


Main-Locus Observation Backends

Section titled “Main-Locus Observation Backends”

std::observe::Scrape, std::observe::Probes, and std::observe::Otlp are declared as params on the main locus and scheduled via placement { }. The compiler recognizes each backend by its type:

Prometheus Metrics Scrape (std::observe::Scrape)

Section titled “Prometheus Metrics Scrape (std::observe::Scrape)”
main locus App {
params {
metrics: std::observe::Scrape = std::observe::Scrape { port: 9100 };
}
placement { metrics: pinned; }
}
fn main() { App { }; }

Health and Capacity Probes (std::observe::Probes)

Section titled “Health and Capacity Probes (std::observe::Probes)”
main locus App {
params {
probes: std::observe::Probes = std::observe::Probes { port: 9090 };
}
run() {
while !self.draining { std::time::sleep(100ms); }
}
}
fn main() { App { }; }

Probes binds <port> on all network interfaces (0.0.0.0) and runs on its own dedicated runtime thread (no placement entry is needed or applied):

Path HTTP Status & Payload Description
/healthz 200 ok or 503 Returns 503 naming any queue that has held unprocessed work for more than 10 seconds (configurable via LOTUS_PROBES_STALL_MS)
/readyz 503 starting, 200 ready, or 503 draining Returns 503 starting until all main locus births complete, 200 ready during normal execution, and 503 draining once drain begins
/capacity 200 (application/json) Reports waiting messages, total capacity, and stalled duration for every runtime queue

Probes does not serve /metrics. For multi-process supervision details, see Multi-binary systems and spec/semantics.md § Probes.

Lifecycle Events Exporter (std::observe::Otlp)

Section titled “Lifecycle Events Exporter (std::observe::Otlp)”
main locus App {
params {
events: std::observe::Otlp = std::observe::Otlp {
endpoint: "http://otel-collector:4318/v1/logs",
};
}
placement { events: pinned; }
}
fn main() { App { }; }

Combining Backends & Zero-Overhead Guarantee

Section titled “Combining Backends & Zero-Overhead Guarantee”

Next: Part III: Services & Concurrency.