feat: add HIL diagnostics and meter health handling

docs/engineering/hil-test-matrix.md

@@ -0,0 +1,20 @@
+# DD3 HIL test matrix
+
+| Area | Stimulus | Required evidence | Expected result |
+|---|---|---|---|
+| Boot | Flash/start both boards | `boot`, `role`, periodic `health` from each | Exactly one sender and receiver; no reset loop; bounded heap/stack |
+| Time | Sender receive window and receiver time-sync | sender request/completion, receiver transmit | Valid UTC completion; misses remain visible |
+| Meter baseline | Real optical meter | `meter_frame`, `meter_sample` | Frame classification and seconds index when present; no secrets/values logged |
+| Batch | Normal 30 s batching | create, encode, every TX/RX chunk, complete, decode, ACK TX/RX | One batch ID correlates end to end |
+| Meter timeout | Partial start then >1.5 s pause | `classification=timeout`, health | No reset; next valid fixture recovers |
+| Meter malformed | Each parser fixture then valid | failure/partial classification, then valid | No reset; failure is not treated as valid evidence |
+| Meter time | rollback/jump seconds fixtures | `meter_time` rollback/jump | Diagnostic classification; subsequent valid frame recovers |
+| Oversize/slow | >512-byte or slow fixture | oversized or valid event, health | Bounded buffers and continued operation |
+| Drop chunk | `drop_chunk` once | incomplete transfer, retry, later complete/ACK | Incomplete batch is not decoded; retry recovers |
+| Duplicate chunk | `duplicate_chunk` once | duplicate/incomplete diagnostic, retry, later ACK | No double publication; transfer recovers |
+| Corrupt chunk | post-CRC byte flip once | `lora_reject=crc_fail`, retry, later ACK | CRC rejects frame; incomplete batch not decoded |
+| Suppress ACK | suppress one ACK response | receiver suppression, sender timeout/retry, later ACK | Retry count increments and recovers |
+| Delay ACK | delay one ACK response | delay event and ACK/retry timing | No crash/deadlock; transfer recovers |
+| Wrong ACK ID | altered batch ID once | sender rejected `ack_rx`, retry, correct ACK | Incorrect ACK cannot complete inflight batch |
+
+For every fault case, compare `health` events before and after injection, require a later successful ACK, and inspect raw logs for resets. MQTT duplicate-publication evidence may require a separately approved broker observer; without it, report that assertion as blocked rather than inferring success solely from radio logs.

docs/engineering/local-hil-setup.md

@@ -0,0 +1,94 @@
+# Safe local DD3 HIL setup
+
+## Safety boundary
+
+This harness is for two USB-connected LilyGO boards and, optionally, an electrically isolated USB-UART or optical meter simulator. It does not authorize direct connection to mains-referenced meter conductors. Do not change wiring, GPIO14 role selection, straps, eFuses, secure boot, flash encryption, NVS, credentials, or udev/OS permissions as part of this workflow.
+
+The firmware remains role-neutral: GPIO14 is sampled at boot exactly as before. The `hil` environment inherits `lilygo-t3-v1-6-1` (the current 433 MHz debug build) and adds only `ENABLE_HIL_TRACE=1`. Production environments are unchanged.
+
+## Dependencies and Linux serial access
+
+PlatformIO is accepted from `PATH` or `~/.platformio/penv/bin/pio`. Install Python dependencies into a repository-local environment:
+
+```bash
+python3 -m venv .venv-hil
+.venv-hil/bin/pip install -r requirements-hil.txt
+```
+
+Inspect access with `id -nG` and `stat -c '%A %U %G %n' /dev/ttyACM0`. On this machine the user is already in `dialout`, and the discovered ports are `root:dialout` with group read/write. The tools never run `sudo`, modify group membership, change permissions, or install udev rules. If access is absent, stop and ask the system administrator to apply the site's normal policy, then start a new login session.
+
+## Discover and approve stable ports
+
+Discovery lists metadata and symlinks but does not open a device:
+
+```bash
+.venv-hil/bin/python tools/hil/discover.py
+```
+
+Prefer `/dev/serial/by-id` names. Copy `.hil/local.example.toml` to ignored `.hil/local.toml` only after visually confirming the two targets. Set the sender and receiver path, then set `approved_access = true` only after approving the first serial open. Set `approved_flash = true` only after separately approving flashing. The tools refuse missing, duplicate, unstable, or unapproved ports. Never commit local `/dev` paths.
+
+Current discovery on 2026-07-13 found two uniquely identified board candidates and one ambiguous device:
+
+```text
+usb-1a86_USB_Single_Serial_5644005006-if00 -> ttyACM0
+usb-1a86_USB_Single_Serial_5644020877-if00 -> ttyACM1
+usb-1a86_USB_Serial-if00-port0             -> ttyUSB0 (ambiguous; do not access)
+```
+
+Serial role evidence, not the filename, is authoritative. A valid mapping requires a `HIL:` role event from both devices and must agree with the configured names.
+
+## Build, flash, and capture
+
+Building never accesses serial devices and does not require local port configuration:
+
+```bash
+.venv-hil/bin/python tools/hil/build.py
+```
+
+After explicit access and flash approval is recorded:
+
+```bash
+.venv-hil/bin/python tools/hil/flash_pair.py --yes-flash
+.venv-hil/bin/python tools/hil/capture_pair.py --reset --seconds 180
+.venv-hil/bin/python tools/hil/run_baseline.py --yes-flash --seconds 180
+```
+
+`flash_pair.py` builds the HIL environment once, then uploads the same environment to each approved stable path. It does not issue a full-chip or NVS erase; normal uploading necessarily rewrites the bootloader, partition-table, boot-app, and application address ranges shown in the flash log. `--reset` pulses the boards' normal USB auto-reset circuit only after capture readers are active. Ctrl+C closes both serial handles and retains all bytes captured so far.
+
+Each run writes beneath `artifacts/hil/<UTC timestamp>-<label>/`: exact per-device `.raw` streams, timestamped `.log` streams, merged `events.jsonl`, build/flash logs, `report.md`, and `junit.xml`. A report fails unless device evidence correlates a sender batch ID through receiver reassembly, decode, ACK transmission, and sender ACK reception.
+
+## Meter simulator
+
+Configure `ports.meter_simulator` only for a separately identified and approved isolated USB-UART/optical adapter connected to the board's safe optical UART boundary. The simulator uses `9600 7E1`:
+
+```bash
+.venv-hil/bin/python tools/hil/meter_simulator.py valid
+.venv-hil/bin/python tools/hil/meter_simulator.py malformed_then_valid
+```
+
+Available fixtures cover timeout, truncation, missing terminator, malformed/missing OBIS, invalid/rollback/jump meter seconds, oversize, slow bytes, and malformed-then-valid recovery. Do not point this configuration at either LilyGO console port or an unidentified adapter.
+
+## One-shot LoRa fault injection
+
+Fault hooks exist only in the `hil` build, are disabled at boot, and are consumed once. The capture tool arms them through the already approved console after the requested delay:
+
+```bash
+.venv-hil/bin/python tools/hil/capture_pair.py --seconds 180 --fault drop_chunk --fault-after 40
+.venv-hil/bin/python tools/hil/run_baseline.py --yes-flash --fault suppress_ack --seconds 240
+.venv-hil/bin/python tools/hil/run_fault_suite.py --seconds 600
+```
+
+Supported hooks: drop, duplicate, or post-CRC corrupt one chunk; suppress or delay one ACK response; and send one ACK response with a wrong batch ID. Clear pending hooks with `HILCMD:clear`. Production builds contain no active hook implementation.
+
+## Interpret and restore
+
+Regenerate reports with:
+
+```bash
+.venv-hil/bin/python tools/hil/report.py artifacts/hil/<timestamp>
+HIL_ARTIFACT=artifacts/hil/<timestamp> .venv-hil/bin/pytest -v -m hil hil_tests/
+```
+
+`FAIL` means required device evidence is missing or contradictory. Meter evidence becomes `BLOCKED` when a real meter or approved isolated simulator is silent/unavailable. Never convert a blocked or missing-evidence result into a pass manually.
+
+To restore normal firmware, use the correct existing production environment for the board's radio (`lilygo-t3-v1-6-1-prod` or `lilygo-t3-v1-6-1-868-prod`) and the same approved flash procedure/port boundary. Reconfirm the environment and both target paths before uploading; restoring production is also a flash operation and requires explicit approval.