Testing pipeline

Reference material, synced from the compiler repo’s spec/. The guide is the gentler path in.

The testing pipeline is part of the language toolchain, not an add-on. hale test ships in the same binary as hale build. Test infrastructure exists from day 1 because the language’s discipline (closure tests, k_max bounds, projection-class invariants, multi-perspective stability commit-rules) needs testing infrastructure to be enforced.

hale test (Layer 1 + Layer 2), hale bench (Layer 3 single-language), hale fmt, and hale doc all ship in the CLI today (lex / parse / check / run / build / test / bench / verify / fmt / doc / fetch / lsp / mcp). Only hale bench -compare remains design-only below. hale verify runs check’s exact analysis surface but GATES: any finding — advisory or error — exits 1, making it the CI discipline gate (where check stays the fast advisory oracle: warnings print, only errors fail).

Hale testing distinguishes three layers, each with its own tooling:

Does this program parse, typecheck, and have the meaning the language spec says it should?

• Parser tests. Given source, the parser produces the expected AST (or rejects with the expected error). Stored as .hl files paired with .expected.json (or similar) AST dumps. Driven by the grammar in spec/grammar.ebnf.
• Typechecker tests. Given a program, the typechecker accepts or rejects with the expected diagnostic. Same shape as parser tests.
• Operational-semantics tests. Given a program and inputs, the program produces the expected output. Driven by the operational semantics document (yet to be written).

These run as part of hale test and as part of compiler CI. A compiler regression should be caught here.

Does the framework’s discipline hold for this program?

The language’s job is not just “compile the source” — it’s “refuse to compile a program that violates framework discipline.” The framework’s commitments need test infrastructure:

• k_max bound verification. For every locus, the compiler computes k_max = B / [(1 − phi) * c + phi * sigma] and checks that no accept call site can exceed it. Tests assert the compiler rejects over-budget call sites.
• Closure-test existence. For every closure name { ... } block, the compiler verifies the cycle exists (both sides of ~~ reference defined values within the same scope). Tests assert the compiler rejects cycles that don’t close.
• Projection-class invariants. A locus declared projection rich cannot be instantiated with N > rich’s bound; etc. Tests assert mismatches are rejected.
• Multi-perspective stability commit-rules. A perspective with stable_when |perspectives| >= 3 cannot be serialized with fewer perspectives validated. Tests assert violations fail at runtime.
• Substrate-derivation discipline. When a value carries anchor metadata, anchor-self-consistent uses are flagged. Tests assert anchor-self-consistency triggers a warning or error per declared policy.
• Vertical-only flow. Lateral references between sibling loci are compile-time rejected. Tests assert the compiler rejects sibling-to-sibling access.

These tests are written in Hale itself. The standard library provides assert(...), assert_rejects(...), assert_closure(...) and similar primitives. Test programs are valid Hale programs; the framework discipline applies to them too.

Does this program meet its declared performance envelope? And how does it compare to equivalent implementations in other languages?

A benchmark is a function annotated with bench (TBD: grammar extension, or stdlib function with a magic name like Go’s Benchmark*). The runner invokes it for a measured number of iterations; reports time-per-op, allocations-per-op, memory high-water.

fn bench_hello() {
    Hello { };
}

Output is JSON-serializable for CI consumption. Baselines are checkpointed in version control; regressions produce a diff the developer must explicitly accept.

These are internal development tools, not published results. Their purpose is to give the team visibility into the language’s performance shape as it evolves — to catch regressions, validate that framework-discipline overhead is in the expected range, and spot when a design choice is costing us order-of-magnitude throughput.

A benchmark file declares its equivalent in another language as a sibling:

//
// @external_equivalents:
//   - lang: go
//     path: ./equivalents/message_passing.go
//   - lang: rust
//     path: ./equivalents/message_passing.rs

The runner builds and runs each; reports a comparison table. The author writes the equivalent however they think is fair for the comparison they want — there is no “fairness review,” because nothing is being published. The numbers are useful to us; they don’t need to be defensible to outsiders.

Useful comparative-perf categories for internal use:

• Coordination-overhead. Many-message-passing scenarios vs. Erlang / Go.
• Region-allocation throughput. Allocation / deallocation rate vs. GC’d languages.
• Closure-test overhead. Same program with closure tests on / off — measures the cost of the framework discipline.
• Mode-projection. Same kernel computed three ways (bulk / harmonic / resolution) vs. a hand-written per-N-implementation in another language.

Comparative results are not gatekept; any branch can produce them and stash them in bench-results/ (gitignored). A regression in hale-vs-X ratio is a developer signal, not a CI gate.

Every benchmark has a stored baseline (numerical envelope, not a fixed value — a tolerance band). The runner asserts current runtime is within band. Bands tighten over time as the compiler improves; widening a band is an explicit, reviewed action.

project/
├── src/
│   └── *.hl            // production source
└── tests/
    ├── unit/
    │   └── *_test.hl   // unit tests, by module
    ├── integration/
    │   └── *_test.hl   // multi-locus integration tests
    ├── bench/
    │   └── *_bench.hl  // benchmarks
    └── equivalents/    // external-language equivalents for
        ├── go/         // comparative benchmarks
        ├── rust/
        └── erlang/

Or, alternatively, Go-style: *_test.hl lives next to the source it tests. Both layouts are supported; the runner finds tests by suffix (_test.hl) regardless of location.

hale test runs Layer 1 + Layer 2 today; hale bench runs Layer 3’s single-language half.

hale bench [file | dir] discovers *_bench.hl files (dir walk, vendor/ and dot-dirs skipped); every zero-param free fn named bench_* is a benchmark. The runner appends a synthesized driver main (a bench file must not define its own), compiles at the release profile with the same hale.toml [ffi] pickup as build/test, and runs it. The driver self-calibrates Go-style: batch sizes grow ×10 until one batch takes ≥100 ms, then the final batch reports ns/op and allocs/op (std::diag::heap_alloc_count deltas; shown as - in sanitizer builds where the counting shim is absent). -run <substr> filters by bench name; --json emits one record per bench (file/name/iters/ns_per_op/allocs_per_op) for CI. Benchmarks may print their own output — non-report lines pass through.

Still planned from the original design: stored baselines with tolerance bands as a CI gate, and -compare external-language equivalents. The fn bench_* magic-name convention (Go’s Benchmark* shape) is the resolved answer to the “grammar extension or magic name?” question above — no grammar change.

Zero config. hale doc [file | dir] renders a seed’s API reference from /// doc comments (the convention in spec/tokens.md): every public top-level declaration — fns, loci (with params and their documented methods), types, topics, interfaces, consts — with its signature and doc text. Markdown to stdout by default; -o <path> writes it; --json emits one record per declaration (file/kind/name/signature/doc/ members) for tooling and agents. __-prefixed names and main are internal and skipped; a file that doesn’t parse is reported and skipped with exit 1. Doc text is recovered positionally (the lines directly above the declaration, stepping over decorator lines), so the lexer and AST are untouched.

hale doc --stdlib renders the std:: surface instead: the rename table supplies public paths, the bundled stdlib source supplies decl shapes + /// docs (mangled param types demangled; internal-typed params hidden), and the typecheck signature table supplies the C-primitive-backed free fns that have no .hl decl. The spec/stdlib.md tables remain the canonical CONTRACT; the generated reference is the browsable companion, and stdlib declarations grow /// docs namespace-by-namespace (metrics, log, and BytesBuilder are done).

Zero config, Go-style: there are no options that change the output. hale fmt [paths] formats .hl files in place (no path = the current directory tree; vendor/ and dot-directories are skipped); --check lists files that would change and exits 1 (the CI gate); --diff previews without writing; --stdin filters stdin→stdout for editor integration.

What canonical form means (a token-stream formatter — the author’s line-break structure is PRESERVED, gofmt-style; there is no max-line-length enforcement):

• Indentation — 4 spaces per bracket depth. A closing bracket returns to its opener’s line indent; brackets opened together on one line indent their contents once. Bracket-less continuation lines (a leading &&/., a trailing binary operator on the previous line) get one extra level.
• Spacing — canonical pair rules: binary operators spaced, unary -/! tight to their operand, ./::/.. tight, nothing inside ( ) [ ], literal braces spaced (Rec { key: 1 }, { }), : tight-left (except the spaced locus X : serves P conformance colon, per this spec’s own examples), generic angles tight (Holder<Int>), lifecycle parens tight (run()).
• Blank lines — collapsed to at most one; none at file start; exactly one trailing newline. Intra-line alignment padding (let x = 1;) collapses to single spaces.
• Comments — preserved verbatim in position: own-line comments indent with the code, trailing comments sit one space after it.

Safety: the formatter re-lexes its own output and refuses to write unless the semantic token stream is byte-identical to the input’s — a formatter bug can mangle whitespace, never what the compiler sees. Files that don’t lex are reported and left untouched. Formatting is idempotent; the corpus test (hale-syntax/tests/fmt_corpus.rs) holds every fixture example and stdlib source to both properties.

Provided by std::test. Not a separate testing framework; the language’s stdlib includes test primitives.

Three primitives, all written purely in Hale (composing std::process::exit):

fn main() {
    std::test::assert(2 + 2 == 4, "trivial arithmetic");
    std::test::assert_eq_int(answer(), 42, "answer");
    std::test::assert_eq_str(greet("world"), "hello, world", "greeting");
}

The test-runner contract is exit-code based:

• Pass = exit 0 with no stdout. A test program that runs to completion silently has passed.
• Fail = non-zero exit code with ASSERTION FAILED: <msg> (and, for assert_eq_*, expected: X / actual: Y) on stdout. The first failure short-circuits — std::process::exit terminates immediately.

A .hl test program is just an ordinary Hale binary.

tests/hale/ holds *_test.hl programs that test the language in the language, run by hale test and wired into the workspace suite (crates/hale-cli/tests/hale_native_suite.rs). Run them directly with hale test tests/hale.

A behaviour test belongs there rather than in a Rust integration test when it is “run a program, check what it computed”. Two things change in the move:

• The expectation stops being transcribed. The Rust form compiles a program that prints, then substring-matches the output from another language. The Hale form asserts next to the code — no second copy to drift. It is also stricter: assert_eq_int(n, 42) rejects what stdout.contains("a=42") accepts, because the latter also passes on a=421.
• The program gets typechecked. build_executable, which the Rust codegen tests call, parses and lowers but never runs the checker, so those programs are compiled and executed without ever being checked.

What stays in Rust is anything asserting on compiler output rather than program behaviour: diagnostics, IR shape, leak counts, observation records.

Single-pool execution is deterministic. A program whose loci all run on the main cooperative scheduler — no placement pinning, no additional pools — produces the same publishes and the same deliveries in the same order on every run, given the same inputs. This is a guarantee, not an observation: one pool is one consumer thread by construction (spec/runtime.md § Scheduler — the same invariant dispatch devirtualization rests on), handlers run to completion, and the main-thread bus queue drains FIFO, so there is no scheduling freedom for an order to vary within.

“Given the same inputs” is load-bearing: time, entropy, environment, FFI returns, and socket reads are inputs, and a program that consumes them may compute differently even though its scheduling cannot reorder. Whether a body reaches those inputs is a compile-time question — @deterministic (spec/verification.md § Effect assertions) asserts it, and frontier::infer_effects answers it without any annotation.

Pinned: crates/hale-codegen/tests/replay_determinism.rs compares the complete ordered BUS_PUBLISH/BUS_DELIVER record sequence of a multi-locus cascade across repeated runs. A divergence there is a runtime bug, never a flaky test.

Multi-pool programs get no ordering guarantee across pools today: each pool’s single consumer thread has a well-defined per-consumer delivery order, but the interleaving between pools is OS scheduling. Recording that per-consumer order (and replaying it) is the record/replay track — GH #296.

Reserved as a future extension. The language’s strong type-and-discipline surface makes property-based testing particularly natural — you can declare properties that should hold for all inputs and let the runner generate counter-examples. Not in the v0.1 stdlib.

The toolchain emits machine-readable output:

• hale test --json produces JSON test results (per-test pass/fail, per-test timing, error messages).
• hale bench --json produces JSON benchmark output (per-bench time, allocations, comparative table if -compare given).
• hale check --json produces JSON diagnostics.

CI consumes the JSON; standard reporters (JUnit XML, GitHub Actions annotations, etc.) are downstream conversions.

1. bench annotation: keyword, attribute, or naming convention? Go uses BenchmarkName. Rust uses #[bench]. Hale has neither attributes nor magic-name conventions yet. Decision pending; probably an attribute (@bench fn ...) added to the grammar in v0.2.
2. Determinism. Does the runtime need deterministic scheduling for benchmark consistency? Resolved for the single-pool case: it already has it, by construction — see § Determinism above. Deterministic re-execution of multi-pool programs is the record/replay track (GH #296); forcing deterministic scheduling in production is an explicit non-goal there.
3. External-language toolchain access. hale bench -compare needs go, rustc, erlc, etc. on PATH. Documenting this clearly is dev-experience work.