Multi-Binary Deployment & Fleets

In Hale, splitting a single-binary program into multiple communicating services requires zero changes to publisher or subscriber code. By adding a bindings { } block to main locus, you bind in-process topics to Unix sockets, shared-memory rings, or custom broker adapters, and use a fleet locus to verify architectural claims and render deployment manifests across the entire multi-binary topology.

type Match { size: Int; }
topic MatchReady { payload: Match; }

locus Matchmaker {
    params { target_size: Int = 2; }
    bus { publish MatchReady; }
    run() { MatchReady <- Match { size: self.target_size }; }
}

main locus App {
    bindings {
        MatchReady: unix("/tmp/matches.sock");
    }
    run() {
        Matchmaker { target_size: 4 };
    }
}

fn main() { App { }; }

Binding Topics to Transports (bindings { })

When a topic is not listed in bindings { }, the runtime delivers its messages through an in-process queue. Declaring the topic inside main locus’s bindings { } block routes it across processes while keeping Topic <- payload; and subscribe Topic as handler; unchanged.

Transport	Syntax	Delivery Contract
In-process (default)	(omitted from bindings)	Dispatched to every live subscriber inside the current process.
Unix domain socket	unix("/path", role: listen | connect)	Framed AF_UNIX byte stream. Role is inferred from whether the binary subscribes (listen, up to 64 concurrent publishers) or publishes (connect); specify role: explicitly when one binary does both.
Shared-memory ring	shm_ring("/name", on_overflow: ...)	Lock-free POSIX shared-memory ring for high-frequency same-host IPC. Slot claimed and committed under on_overflow policy. See Zero-Copy Bus.
Custom adapter locus	MyAdapter { ... }	Any user or library locus implementing fn send(subject: String, bytes: Bytes) (__StdBusAdapter), such as NATS, MQTT, or WebSocket framing.
Operator UDP route	udp://host:port (in LOTUS_BUS_CONFIG)	Configured at deploy time via LOTUS_BUS_CONFIG (generated by hale node), not in source. Supports IPv4 multicast; best-effort/lossy by design.

Note

For streaming topics to external CLI or dashboard clients or exposing RPC endpoints, see The API Surface (hub bindings and api::serve).

Cross-Seed Topics & Keyed Routing

• Binding imported topics: bindings { } can reference topics imported from shared library seeds (core::Verdict: unix("/tmp/verdicts.sock", role: listen);).
• Keyed routing preserved: Topics declared with keyed_by retain their routing behavior across unix sockets and custom adapters. The receiving process decodes incoming bytes once (through the binding’s codec, if present), extracts the routing key from the payload, and delivers the message only to matching where key == ... subscribers.
• Orphan topic analysis with __local_dispatch: Custom adapters inject inbound network frames into the local bus by calling std::bus::__local_dispatch(subject, bytes). hale check counts __local_dispatch with a literal subject as a publisher of that topic, and __local_dispatch with a dynamic subject string as a potential publisher of any subscribed topic in the binary.

Writing Custom Transport Adapters

A custom adapter is an ordinary locus that provides fn send(subject: String, bytes: Bytes):

locus BrokerAdapter {
    bus { publish Outgoing; subscribe Outgoing as on_outgoing; }
    fn send(subject: String, bytes: Bytes) {
        Outgoing <- Frame { subject: subject, data: bytes };
    }
    fn on_outgoing(f: Frame) { /* write frame to socket on adapter thread */ }
    run() { while !self.draining { /* read socket and call std::bus::__local_dispatch */ } }
}

bindings {
    BrokerEvt: nats::NatsAdapter { url: "nats://prod:4222" } codec(nats::JsonCodec { });
}

Adapter loci follow three rules:

1. Constructed before main: The adapter is instantiated once before main locus. Its literal cannot read self.<param> of main, execute terminate;, or publish to the bus during construction.
2. Pinned receive thread: The adapter runs on its own dedicated pinned OS thread. Its run() loop and any bus handlers it subscribes to execute on that thread, whereas send(subject, bytes) is called directly on each publisher’s thread. (To perform socket writes exclusively on the adapter’s thread, have send publish onto an internal topic that the adapter subscribes to, as shown above.)
3. Pinned lifecycle constraints: Like any pinned locus, an adapter cannot declare accept(...) or closures with birth or dissolve epochs (epoch inline closures are supported).

Connection Lifecycle, Supervision, and Teardown

Startup Verification & Connect Supervision

A broker never accepts messages onto a broken transport:

• Boot failure on unreachable binding: If any bindings { } entry or LOTUS_BUS_CONFIG route cannot be opened at startup (missing socket directory, port conflict, or exhausted connect retries), the binary exits non-zero at birth with a diagnostic naming the subject.
• Disconnect handling on listen sockets: When a publishing peer disconnects from a unix listener, the listener stays open and accepts subsequent connections seamlessly.
• Supervising connect sockets (std::bus::UnixTransport): If a connect peer drops mid-run, the publishing process exits structurally by default. To reconnect automatically instead, add an on_failure handler for std::bus::UnixTransport on main locus:

type Reading { v: Int; }
topic Evt { payload: Reading; }

main locus App {
    bindings { Evt: unix("/tmp/evt.sock", role: connect); }
    bus { publish Evt; }
    on_failure(t: std::bus::UnixTransport, err: ClosureViolation) {
        restart(t);      // re-runs connect-with-retry
    }
    run() { Evt <- Reading { v: 1 }; }
}

fn main() { App { }; }

Blocking During Reconnect (or wait)

While a supervised UnixTransport is reconnecting, publishes drop by default (incrementing dropped_lost in LOTUS_BUS_COUNTERS_DUMP=1). To park the publisher until the connection is restored, append or wait at the send site:

Evt <- reading or wait;

• Each send site on a topic can choose independently between the default drop behavior and or wait (tracked by the waits counter in LOTUS_BUS_COUNTERS_DUMP=1).
• The single send that initially detects a broken pipe still triggers the structural failure so on_failure can start the reconnect; or wait parks subsequent sends during the reconnect window.
• If reconnection fails permanently or the program begins shutting down, or wait aborts with an error rather than hanging indefinitely.

Listener Buffering & Teardown Order

• Early-arrival buffering (buffered_early): If a remote publisher connects and sends messages in the brief window after a listen socket binds but before local subscribers finish registering, the listener buffers up to 64 messages / 1 MiB per binding and delivers them as soon as registration completes.
• Shutdown quiesce (LOTUS_BUS_QUIESCE_MS): When main returns, listen sockets stop accepting new peers and drain any unread kernel socket buffers for up to 500 ms (configurable via LOTUS_BUS_QUIESCE_MS) while local subscribers are still live.
• Binding teardown order: Transport bindings are initialized in fn main’s prelude before any user code and are dissolved last at shutdown:
  1. Quiesce listen sockets (LOTUS_BUS_QUIESCE_MS).
  2. Join cooperative worker pools.
  3. Drain and dissolve user loci innermost-first (ensuring messages published inside dissolve() hooks still reach the wire).
  4. Dissolve transport bindings, closing sockets and joining transport threads.

Wire Codecs (codec(...))

By default, bound topics serialize payloads using Hale’s internal binary format. To interoperate with non-Hale services over JSON, Protobuf, or custom wire encodings, attach a codec(...) locus to the binding:

bindings {
    Tick: unix("/tmp/ticks.sock") codec(TickJsonCodec { });
    Feed: BrokerAdapter { url: "broker://localhost" } codec(TickJsonCodec { });
}

A codec locus is checked structurally against the topic’s payload type (encode accepts the payload type and returns Bytes/String; decode accepts wire bytes and returns the payload type) and must be stateless (pure) so it can execute safely on transport threads.

Multi-Binary Verification (fleet locus)

While hale check verifies each binary in isolation, a fleet locus models an entire multi-binary deployment—declaring which component binaries run, their host or replica placement, the transport bindings connecting them, and cross-binary architectural claims { }:

import "../desk" as desk;
import "../venue" as venue;
import "../shared" as t;

fleet locus Prod {
    params {
        orders: desk::Orders;
        gw: venue::Gateway;
    }
    placement {
        orders: host("box-a");
        gw: replicas(2);
    }
    bindings {
        t::OrderRequest: unix("/run/orders.sock");
    }
    claims {
        one_desk: count publishers(topic t::OrderRequest) == 1;
        no_return: forbid reaches(gw, orders);
    }
}

How Fleet Composition Works

• Artifact-level composition: A fleet seed never merges source ASTs across binaries (which would conflate in-process topics with cross-process routes). Instead, hale fleet check compiles each referenced seed into its admitted ComponentModel artifact and joins them strictly along the topics declared in the fleet’s bindings { } block.
• Process instances (<param>-<i>): Each parameter expands into concrete process instances based on its placement (orders-0 on box-a; gw-0 and gw-1 for replicas(2)), allowing claims to reason about exact publisher/subscriber cardinalities across replicas.
• Binding consistency checks: If an individual binary’s main locus also declares a binding for a fleet topic, both must specify the same transport. Any fleet binding lacking a publisher or a subscriber fails verification.

Register fleet seeds in hale.toml to verify all environments together:

[fleets]
production = "fleets/prod"
staging    = "fleets/staging"

Fleet CLI Commands & Cryptographic Signing

Command	Description
hale fleet check <fleet-dir>	Extracts component artifacts, derives cross-binary routes, and verifies all fleet claims { }.
hale fleet export <fleet-dir>	Exports the resolved instances, seed hashes, hosts, derived routes, and claims as JSON.
hale fleet keygen <name>	Generates an ES256 signing keypair (<name>.pem and <name>.pub.pem).
hale fleet sign <fleet-dir> --key <name>.pem	Signs the exact fleet export bytes, writing <fleet-dir>/fleet.sig.
hale fleet check <fleet-dir> --trust <name>.pub.pem	Verifies fleet.sig (and component artifact hashes) before checking claims. Also configurable via [fleet_trust] keys = [...] in hale.toml.

When a fleet claim fails, hale fleet check prints the exact cross-binary witness path:

fleet claim `orders_pass_oms` violated — witness:
  prober-0::Probe::submit  [prober/main.hl]
  -(route `t::OrderRequest`)->
  gw-0::Gateway::on_order  [gw/main.hl]

Note

What Fleet Verification Certifies hale fleet check proves static route topology and capability claims across compiled artifacts. It does not prove runtime network delivery (which depends on the transport protocol and live peers) or compatibility of mixed versions during a rolling upgrade. If a forbidden path encounters an unresolvable indirect call, the claim returns uncertified and fails the build.

Platform Health Probes (std::observe::Probes)

To expose HTTP health and readiness endpoints for container orchestrators (Docker Compose, Kubernetes) even on binaries that serve no user-facing HTTP API, add std::observe::Probes to main locus’s params:

main locus App {
    params {
        probes: std::observe::Probes = std::observe::Probes { port: 9100 };
    }
    run() {
        while !self.draining { std::time::sleep(100ms); }
    }
}
fn main() { App { }; }

The runtime spawns a dedicated probe thread (isolated from application cooperative pools so a wedged pool never blocks health checks) serving three endpoints:

Endpoint	Responses & Semantics
GET /healthz	Returns 200 ok, or 503 naming any queue (main or cooperative pool) where queued work has gone untouched for over 10 seconds (configurable via LOTUS_PROBES_STALL_MS). An idle queue or a single long handler with an empty queue remains 200 ok.
GET /readyz	Returns 503 starting while main’s children are birthing and bindings are opening, 200 ready during steady state, and 503 draining as soon as shutdown begins.
GET /capacity	Returns a JSON snapshot of each queue’s waiting cell count, maximum capacity, and current stall duration.

std::observe::Probes must be initialized via its parameter default on main locus (the port may reference an earlier parameter, e.g., port: self.probe_port, but callers constructing main cannot pass a custom Probes instance).

Rendering Deployment Manifests (hale deploy render)

hale deploy render translates a seed or fleet locus into Docker Compose or Kubernetes manifests without building images or touching live infrastructure:

hale deploy render fleets/prod --env prod
hale deploy render fleets/prod --backend compose --placement alloc.json --out deploy/
hale deploy render ./svc --backend kubernetes --out deploy/

Configure target environments in hale.toml:

[environments.prod]
source_only = true
backend = "compose"

[environments.prod.env]
LOG_LEVEL = "info"

• Per-host manifest generation: For a fleet locus, hale deploy render outputs one manifest per target host (<out>/<host>/). You can override static host placements by passing a JSON allocation file ([{"instance": "gw-0", "host": "box-a"}, {"instance": "gw-1", "host": "box-b"}]) via --placement.
• Compose vs. Kubernetes mapping:
  • Compose: Renders one service per instance on that host, mounting pre-built binaries (hale build each seed first), adding depends_on for same-host routes, and recording cross-host routes (gw-1@box-b) under x-hale-routes.
  • Kubernetes: Renders one Deployment per program with replicas set to its instance count on that host and annotates cross-host routes in comments.
• Automatic probe and metrics wiring: If a program declares std::observe::Probes, hale deploy render generates the corresponding Compose healthcheck or Kubernetes liveness/readiness probes (using the default value of self.probe_port if parameterized). If a program declares std::observe::Scrape, the manifest publishes the metrics port and adds prometheus.io/* scrape annotations.

Next: Composition Patterns.