feat: add HIL diagnostics and meter health handling

2026-08-17 22:33:04 +02:00
parent bc2ac6e8f6
commit aa25aa2c12
30 changed files with 2108 additions and 236 deletions
@@ -3,3 +3,8 @@
 .vscode/c_cpp_properties.json
 .vscode/launch.json
 .vscode/ipch
+.hil/local.toml
+artifacts/hil/
+__pycache__/
+.pytest_cache/
+.venv-hil/
@@ -0,0 +1,20 @@
+# DD3 HIL test matrix
+
+| Area | Stimulus | Required evidence | Expected result |
+|---|---|---|---|
+| Boot | Flash/start both boards | `boot`, `role`, periodic `health` from each | Exactly one sender and receiver; no reset loop; bounded heap/stack |
+| Time | Sender receive window and receiver time-sync | sender request/completion, receiver transmit | Valid UTC completion; misses remain visible |
+| Meter baseline | Real optical meter | `meter_frame`, `meter_sample` | Frame classification and seconds index when present; no secrets/values logged |
+| Batch | Normal 30 s batching | create, encode, every TX/RX chunk, complete, decode, ACK TX/RX | One batch ID correlates end to end |
+| Meter timeout | Partial start then >1.5 s pause | `classification=timeout`, health | No reset; next valid fixture recovers |
+| Meter malformed | Each parser fixture then valid | failure/partial classification, then valid | No reset; failure is not treated as valid evidence |
+| Meter time | rollback/jump seconds fixtures | `meter_time` rollback/jump | Diagnostic classification; subsequent valid frame recovers |
+| Oversize/slow | >512-byte or slow fixture | oversized or valid event, health | Bounded buffers and continued operation |
+| Drop chunk | `drop_chunk` once | incomplete transfer, retry, later complete/ACK | Incomplete batch is not decoded; retry recovers |
+| Duplicate chunk | `duplicate_chunk` once | duplicate/incomplete diagnostic, retry, later ACK | No double publication; transfer recovers |
+| Corrupt chunk | post-CRC byte flip once | `lora_reject=crc_fail`, retry, later ACK | CRC rejects frame; incomplete batch not decoded |
+| Suppress ACK | suppress one ACK response | receiver suppression, sender timeout/retry, later ACK | Retry count increments and recovers |
+| Delay ACK | delay one ACK response | delay event and ACK/retry timing | No crash/deadlock; transfer recovers |
+| Wrong ACK ID | altered batch ID once | sender rejected `ack_rx`, retry, correct ACK | Incorrect ACK cannot complete inflight batch |
+
+For every fault case, compare `health` events before and after injection, require a later successful ACK, and inspect raw logs for resets. MQTT duplicate-publication evidence may require a separately approved broker observer; without it, report that assertion as blocked rather than inferring success solely from radio logs.
@@ -0,0 +1,94 @@
+# Safe local DD3 HIL setup
+
+## Safety boundary
+
+This harness is for two USB-connected LilyGO boards and, optionally, an electrically isolated USB-UART or optical meter simulator. It does not authorize direct connection to mains-referenced meter conductors. Do not change wiring, GPIO14 role selection, straps, eFuses, secure boot, flash encryption, NVS, credentials, or udev/OS permissions as part of this workflow.
+
+The firmware remains role-neutral: GPIO14 is sampled at boot exactly as before. The `hil` environment inherits `lilygo-t3-v1-6-1` (the current 433 MHz debug build) and adds only `ENABLE_HIL_TRACE=1`. Production environments are unchanged.
+
+## Dependencies and Linux serial access
+
+PlatformIO is accepted from `PATH` or `~/.platformio/penv/bin/pio`. Install Python dependencies into a repository-local environment:
+
+```bash
+python3 -m venv .venv-hil
+.venv-hil/bin/pip install -r requirements-hil.txt
+```
+
+Inspect access with `id -nG` and `stat -c '%A %U %G %n' /dev/ttyACM0`. On this machine the user is already in `dialout`, and the discovered ports are `root:dialout` with group read/write. The tools never run `sudo`, modify group membership, change permissions, or install udev rules. If access is absent, stop and ask the system administrator to apply the site's normal policy, then start a new login session.
+
+## Discover and approve stable ports
+
+Discovery lists metadata and symlinks but does not open a device:
+
+```bash
+.venv-hil/bin/python tools/hil/discover.py
+```
+
+Prefer `/dev/serial/by-id` names. Copy `.hil/local.example.toml` to ignored `.hil/local.toml` only after visually confirming the two targets. Set the sender and receiver path, then set `approved_access = true` only after approving the first serial open. Set `approved_flash = true` only after separately approving flashing. The tools refuse missing, duplicate, unstable, or unapproved ports. Never commit local `/dev` paths.
+
+Current discovery on 2026-07-13 found two uniquely identified board candidates and one ambiguous device:
+
+```text
+usb-1a86_USB_Single_Serial_5644005006-if00 -> ttyACM0
+usb-1a86_USB_Single_Serial_5644020877-if00 -> ttyACM1
+usb-1a86_USB_Serial-if00-port0             -> ttyUSB0 (ambiguous; do not access)
+```
+
+Serial role evidence, not the filename, is authoritative. A valid mapping requires a `HIL:` role event from both devices and must agree with the configured names.
+
+## Build, flash, and capture
+
+Building never accesses serial devices and does not require local port configuration:
+
+```bash
+.venv-hil/bin/python tools/hil/build.py
+```
+
+After explicit access and flash approval is recorded:
+
+```bash
+.venv-hil/bin/python tools/hil/flash_pair.py --yes-flash
+.venv-hil/bin/python tools/hil/capture_pair.py --reset --seconds 180
+.venv-hil/bin/python tools/hil/run_baseline.py --yes-flash --seconds 180
+```
+
+`flash_pair.py` builds the HIL environment once, then uploads the same environment to each approved stable path. It does not issue a full-chip or NVS erase; normal uploading necessarily rewrites the bootloader, partition-table, boot-app, and application address ranges shown in the flash log. `--reset` pulses the boards' normal USB auto-reset circuit only after capture readers are active. Ctrl+C closes both serial handles and retains all bytes captured so far.
+
+Each run writes beneath `artifacts/hil/<UTC timestamp>-<label>/`: exact per-device `.raw` streams, timestamped `.log` streams, merged `events.jsonl`, build/flash logs, `report.md`, and `junit.xml`. A report fails unless device evidence correlates a sender batch ID through receiver reassembly, decode, ACK transmission, and sender ACK reception.
+
+## Meter simulator
+
+Configure `ports.meter_simulator` only for a separately identified and approved isolated USB-UART/optical adapter connected to the board's safe optical UART boundary. The simulator uses `9600 7E1`:
+
+```bash
+.venv-hil/bin/python tools/hil/meter_simulator.py valid
+.venv-hil/bin/python tools/hil/meter_simulator.py malformed_then_valid
+```
+
+Available fixtures cover timeout, truncation, missing terminator, malformed/missing OBIS, invalid/rollback/jump meter seconds, oversize, slow bytes, and malformed-then-valid recovery. Do not point this configuration at either LilyGO console port or an unidentified adapter.
+
+## One-shot LoRa fault injection
+
+Fault hooks exist only in the `hil` build, are disabled at boot, and are consumed once. The capture tool arms them through the already approved console after the requested delay:
+
+```bash
+.venv-hil/bin/python tools/hil/capture_pair.py --seconds 180 --fault drop_chunk --fault-after 40
+.venv-hil/bin/python tools/hil/run_baseline.py --yes-flash --fault suppress_ack --seconds 240
+.venv-hil/bin/python tools/hil/run_fault_suite.py --seconds 600
+```
+
+Supported hooks: drop, duplicate, or post-CRC corrupt one chunk; suppress or delay one ACK response; and send one ACK response with a wrong batch ID. Clear pending hooks with `HILCMD:clear`. Production builds contain no active hook implementation.
+
+## Interpret and restore
+
+Regenerate reports with:
+
+```bash
+.venv-hil/bin/python tools/hil/report.py artifacts/hil/<timestamp>
+HIL_ARTIFACT=artifacts/hil/<timestamp> .venv-hil/bin/pytest -v -m hil hil_tests/
+```
+
+`FAIL` means required device evidence is missing or contradictory. Meter evidence becomes `BLOCKED` when a real meter or approved isolated simulator is silent/unavailable. Never convert a blocked or missing-evidence result into a pass manually.
+
+To restore normal firmware, use the correct existing production environment for the board's radio (`lilygo-t3-v1-6-1-prod` or `lilygo-t3-v1-6-1-868-prod`) and the same approved flash procedure/port boundary. Reconfirm the environment and both target paths before uploading; restoring production is also a flash operation and requires explicit approval.
@@ -0,0 +1,15 @@
+# Local HIL tests
+
+The Python unit tests validate trace parsing, simulator fixtures, and strict evidence correlation without opening hardware. The `hil`-marked test reads an already completed artifact directory; pytest itself never opens or flashes a serial device.
+
+```bash
+python3 -m venv .venv-hil
+.venv-hil/bin/pip install -r requirements-hil.txt
+.venv-hil/bin/pytest -v
+HIL_ARTIFACT=artifacts/hil/<timestamp> .venv-hil/bin/pytest -v -m hil hil_tests/
+HIL_ARTIFACT=artifacts/hil/<fault-timestamp> HIL_FAULT=suppress_ack .venv-hil/bin/pytest -v -m hil hil_tests/
+```
+
+A missing real meter is represented as `BLOCKED`/JUnit `skipped`, not as a firmware failure. A missing boot, role, time-sync, batch, decode, or ACK event is a failure because there is no corresponding device evidence.
+
+Live serial access is performed only by the explicitly invoked tools in `tools/hil/`, after `.hil/local.toml` approval checks.
@@ -0,0 +1,104 @@
+#include "meter_parser.h"
+#include "meter_health.h"
+
+#include <assert.h>
+#include <math.h>
+#include <stdio.h>
+#include <string.h>
+#include <string>
+
+static std::string frame(const char *energy_line = "1-0:1.8.0*255(001234.567*kWh)",
+                         const char *seconds = "00000064",
+                         bool terminator = true) {
+  std::string value = "/DD3HIL\r\n0-0:96.8.0*255(";
+  value += seconds;
+  value += ")\r\n";
+  if (energy_line) {
+    value += energy_line;
+    value += "\r\n";
+  }
+  value += "1-0:16.7.0*255(000950*W)\r\n";
+  value += "1-0:36.7.0*255(000500*W)\r\n";
+  value += "1-0:56.7.0*255(000450*W)\r\n";
+  value += "1-0:76.7.0*255(000000*W)\r\n";
+  if (terminator) value += "!\r\n";
+  return value;
+}
+
+static MeterParseResult parse(const std::string &telegram) {
+  MeterParseResult result = {};
+  meter_parser_parse(telegram.data(), telegram.size(), result);
+  return result;
+}
+
+int main() {
+  MeterHealthState health_state = {};
+  MeterHealthEvaluation health = meter_health_evaluate(health_state, false, UINT32_MAX, 15000);
+  assert(!health.ok && health.fault_started && !health.recovered);
+  health = meter_health_evaluate(health_state, false, UINT32_MAX, 15000);
+  assert(!health.ok && !health.fault_started && !health.recovered);
+  health = meter_health_evaluate(health_state, true, 100, 15000);
+  assert(health.ok && !health.fault_started && health.recovered);
+  health = meter_health_evaluate(health_state, true, 15001, 15000);
+  assert(!health.ok && health.fault_started && !health.recovered);
+
+  MeterParseResult valid = parse(frame());
+  assert(valid.valid);
+  assert(valid.status == MeterParseStatus::Valid);
+  assert(valid.fields_mask == METER_REQUIRED_FIELDS_MASK);
+  assert(valid.meter_seconds_valid && valid.meter_seconds == 100);
+  assert(fabsf(valid.energy_total_kwh - 1234.567f) < 0.001f);
+
+  MeterParseResult wh = parse(frame("1-0:1.8.0*255(1234567*Wh)"));
+  assert(wh.valid);
+  assert(fabsf(wh.energy_total_kwh - 1234.567f) < 0.001f);
+
+  MeterParseResult missing = parse(frame(nullptr));
+  assert(!missing.valid && missing.status == MeterParseStatus::MissingRequired);
+
+  MeterParseResult bad_seconds = parse(frame(nullptr, "GGGGGGGG"));
+  assert(!bad_seconds.valid && bad_seconds.status == MeterParseStatus::InvalidMeterSeconds);
+
+  MeterParseResult bad_unit = parse(frame("1-0:1.8.0*255(1234*J)"));
+  assert(!bad_unit.valid && bad_unit.status == MeterParseStatus::MalformedValue);
+
+  MeterParseResult no_end = parse(frame(nullptr, "00000064", false));
+  assert(!no_end.valid && no_end.status == MeterParseStatus::MissingTerminator);
+
+  MeterParseResult false_prefix = parse(frame("X1-0:1.8.0*255(1234*kWh)"));
+  assert(!false_prefix.valid && false_prefix.status == MeterParseStatus::MissingRequired);
+
+  MeterFrameCollector collector = {};
+  meter_frame_collector_init(collector);
+  uint32_t now = 0;
+  for (char c : std::string("/truncated\r\n")) {
+    meter_frame_collector_feed(collector, c, now += 20);
+  }
+  std::string recovery = frame();
+  assert(meter_frame_collector_feed(collector, recovery[0], now += 20) == MeterFrameEvent::Resync);
+  MeterFrameEvent event = MeterFrameEvent::None;
+  for (size_t i = 1; i < recovery.size(); ++i) {
+    event = meter_frame_collector_feed(collector, recovery[i], now += 20);
+    if (event == MeterFrameEvent::Complete) break;
+  }
+  assert(event == MeterFrameEvent::Complete);
+  MeterParseResult recovered = {};
+  assert(meter_parser_parse(collector.buffer, collector.length, recovered));
+
+  meter_frame_collector_init(collector);
+  meter_frame_collector_feed(collector, '/', 10);
+  assert(meter_frame_collector_check_timeout(collector, 1510, 1500) == MeterFrameEvent::None);
+  assert(meter_frame_collector_check_timeout(collector, 1511, 1500) == MeterFrameEvent::Timeout);
+
+  meter_frame_collector_init(collector);
+  meter_frame_collector_feed(collector, '/', 0);
+  event = MeterFrameEvent::None;
+  for (size_t i = 0; i < METER_FRAME_MAX; ++i) {
+    event = meter_frame_collector_feed(collector, 'A', static_cast<uint32_t>(i + 1));
+    if (event == MeterFrameEvent::Overflow) break;
+  }
+  assert(event == MeterFrameEvent::Overflow);
+
+  puts("meter parser and collector tests passed");
+  return 0;
+}
@@ -0,0 +1,48 @@
+import json
+import sys
+from pathlib import Path
+
+
+ROOT = Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(ROOT / "tools" / "hil"))
+
+from hil_common import parse_hil_line
+from meter_simulator import fixture, frame
+from report import evaluate
+
+
+def test_parse_structured_trace():
+    parsed = parse_hil_line(b'noise HIL:{"event":"boot","ms":12}\r\n')
+    assert parsed == {"event": "boot", "ms": 12}
+    assert parse_hil_line(b'HIL:not-json\n') is None
+
+
+def test_meter_fixtures_are_bounded_and_recoverable():
+    valid = frame()
+    assert valid.startswith(b"/") and b"0-0:96.8.0*255" in valid and b"1-0:1.8.0*255" in valid
+    assert valid.rstrip().endswith(b"!")
+    malformed_then_valid = fixture("malformed_then_valid")
+    assert len(malformed_then_valid) == 2
+    assert malformed_then_valid[-1][0] == valid
+    assert len(fixture("oversized")[0][0]) > 512
+
+
+def test_report_requires_end_to_end_batch_correlation():
+    events = [
+        {"event": "role", "source": "sender", "role": "sender"},
+        {"event": "role", "source": "receiver", "role": "receiver"},
+        {"event": "boot", "source": "sender"},
+        {"event": "boot", "source": "receiver"},
+        {"event": "time_bootstrap", "source": "sender", "stage": "complete", "ok": True},
+        {"event": "meter_frame", "source": "sender", "ok": True},
+        {"event": "health", "source": "sender"},
+        {"event": "health", "source": "receiver"},
+        {"event": "batch_created", "source": "sender", "batch_id": 7},
+        {"event": "reassembly_complete", "source": "receiver", "batch_id": 7},
+        {"event": "payload_decode", "source": "receiver", "batch_id": 7, "ok": True},
+        {"event": "ack_tx", "source": "receiver", "batch_id": 7, "ok": True},
+        {"event": "ack_rx", "source": "sender", "batch_id": 7, "ok": True},
+    ]
+    results = {check.name: check.status for check in evaluate(events)}
+    assert results["batch and ACK correlation"] == "pass"
+    assert all(status == "pass" for status in results.values())
@@ -0,0 +1,54 @@
+import os
+import sys
+from pathlib import Path
+
+import pytest
+
+
+ROOT = Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(ROOT / "tools" / "hil"))
+
+from hil_common import read_events
+from report import evaluate
+
+
+@pytest.mark.hil
+def test_captured_baseline_evidence():
+    artifact_text = os.environ.get("HIL_ARTIFACT", "")
+    if not artifact_text:
+        pytest.skip("set HIL_ARTIFACT to a completed artifacts/hil/<timestamp> directory")
+    artifact = Path(artifact_text)
+    checks = evaluate(read_events(artifact))
+    failures = [f"{check.name}: {check.detail}" for check in checks if check.status == "fail"]
+    assert not failures, "\n".join(failures)
+
+
+@pytest.mark.hil
+def test_captured_fault_recovery_evidence():
+    artifact_text = os.environ.get("HIL_ARTIFACT", "")
+    requested_fault = os.environ.get("HIL_FAULT", "")
+    if not artifact_text or not requested_fault:
+        pytest.skip("set HIL_ARTIFACT and HIL_FAULT to a completed fault-run artifact")
+    checks = evaluate(read_events(Path(artifact_text)), requested_fault)
+    fault_check = next(check for check in checks if check.name == f"fault {requested_fault}")
+    assert fault_check.status == "pass", fault_check.detail
+
+
+@pytest.mark.hil
+def test_meter_outage_is_counted_once_and_not_published():
+    artifact_text = os.environ.get("HIL_ARTIFACT", "")
+    expect_no_meter = os.environ.get("HIL_EXPECT_NO_METER", "") == "1"
+    if not artifact_text or not expect_no_meter:
+        pytest.skip("set HIL_ARTIFACT and HIL_EXPECT_NO_METER=1 for a no-meter capture")
+    events = read_events(Path(artifact_text))
+    sender = [event for event in events if event.get("source") == "sender"]
+    samples = [event for event in sender if event.get("event") == "meter_sample"]
+    active_faults = [event for event in sender
+                     if event.get("event") == "meter_fault" and event.get("state") == "active"]
+    assert samples, "no meter sample evidence captured"
+    assert all(event.get("ok") is False and event.get("classification") == "no_data"
+               for event in samples)
+    assert len(active_faults) == 1, active_faults
+    assert not [event for event in sender if event.get("event") == "batch_created"]
+    assert not [event for event in sender
+                if event.get("event") == "queue" and event.get("action") == "enqueue"]
@@ -0,0 +1,33 @@
+import subprocess
+from pathlib import Path
+
+
+ROOT = Path(__file__).resolve().parents[1]
+
+
+def test_production_meter_parser_and_collector(tmp_path):
+    executable = tmp_path / "meter_parser_test"
+    compile_result = subprocess.run(
+        [
+            "g++",
+            "-std=c++17",
+            "-Wall",
+            "-Wextra",
+            "-Werror",
+            f"-I{ROOT / 'include'}",
+            str(ROOT / "src" / "meter_health.cpp"),
+            str(ROOT / "src" / "meter_parser.cpp"),
+            str(ROOT / "hil_tests" / "meter_parser_harness.cpp"),
+            "-o",
+            str(executable),
+        ],
+        check=False,
+        capture_output=True,
+        text=True,
+    )
+    assert compile_result.returncode == 0, compile_result.stderr
+    run_result = subprocess.run(
+        [str(executable)], check=False, capture_output=True, text=True
+    )
+    assert run_result.returncode == 0, run_result.stderr
+    assert "tests passed" in run_result.stdout
@@ -0,0 +1,41 @@
+#pragma once
+
+#include <Arduino.h>
+#include "config.h"
+
+#ifndef ENABLE_HIL_TRACE
+#define ENABLE_HIL_TRACE 0
+#endif
+enum class HilFault : uint8_t {
+  DropChunk = 0,
+  DuplicateChunk,
+  CorruptChunk,
+  SuppressAck,
+  DelayAck,
+  WrongAckBatch,
+  Count
+};
+
+#if ENABLE_HIL_TRACE
+
+void hil_trace_init(DeviceRole role, uint16_t short_id, int reset_reason);
+void hil_trace_event(const char *event, const char *fields_json = nullptr);
+void hil_trace_eventf(const char *event, const char *fields_fmt, ...);
+void hil_trace_poll_commands();
+bool hil_fault_take(HilFault fault);
+uint32_t hil_fault_delay_ms();
+uint32_t hil_crc32(const uint8_t *data, size_t len);
+void hil_trace_health(uint8_t queue_depth, uint8_t build_depth, uint8_t retry_count);
+
+#else
+
+inline void hil_trace_init(DeviceRole, uint16_t, int) {}
+inline void hil_trace_event(const char *, const char * = nullptr) {}
+inline void hil_trace_eventf(const char *, const char *, ...) {}
+inline void hil_trace_poll_commands() {}
+inline bool hil_fault_take(HilFault) { return false; }
+inline uint32_t hil_fault_delay_ms() { return 0; }
+inline uint32_t hil_crc32(const uint8_t *, size_t) { return 0; }
+inline void hil_trace_health(uint8_t, uint8_t, uint8_t) {}
+
+#endif
@@ -3,7 +3,22 @@
 #include <Arduino.h>
 #include "data_model.h"

+struct MeterDriverStats {
+  uint32_t frames_ok;
+  uint32_t frames_parse_fail;
+  uint32_t rx_overflow;
+  uint32_t rx_timeout;
+  uint32_t bytes_rx;
+  uint32_t last_rx_ms;
+  uint32_t last_good_frame_ms;
+  uint32_t uart_buffer_full;
+  uint32_t uart_fifo_overflow;
+  uint32_t uart_frame_error;
+  uint32_t uart_parity_error;
+};
+
 void meter_init();
 bool meter_read(MeterData &data);
 bool meter_poll_frame(const char *&frame, size_t &len);
 bool meter_parse_frame(const char *frame, size_t len, MeterData &data);
+void meter_get_stats(MeterDriverStats &out);
@@ -0,0 +1,26 @@
+#pragma once
+
+#include <stdint.h>
+
+enum class MeterHealthClassification : uint8_t {
+  Fresh = 0,
+  NoData,
+  Stale
+};
+
+struct MeterHealthState {
+  bool fault_active;
+};
+
+struct MeterHealthEvaluation {
+  bool ok;
+  bool fault_started;
+  bool recovered;
+  MeterHealthClassification classification;
+};
+
+MeterHealthEvaluation meter_health_evaluate(MeterHealthState &state,
+                                             bool has_snapshot,
+                                             uint32_t age_ms,
+                                             uint32_t max_age_ms);
+const char *meter_health_classification_text(MeterHealthClassification classification);
@@ -0,0 +1,54 @@
+#pragma once
+
+#include <stddef.h>
+#include <stdint.h>
+
+constexpr size_t METER_FRAME_MAX = 512;
+constexpr uint8_t METER_REQUIRED_FIELDS_MASK = 0x1f;
+
+enum class MeterFrameEvent : uint8_t {
+  None = 0,
+  Complete,
+  Timeout,
+  Overflow,
+  Resync
+};
+
+struct MeterFrameCollector {
+  char buffer[METER_FRAME_MAX + 1];
+  size_t length;
+  uint32_t last_rx_ms;
+  bool in_frame;
+};
+
+void meter_frame_collector_init(MeterFrameCollector &collector);
+MeterFrameEvent meter_frame_collector_check_timeout(MeterFrameCollector &collector,
+                                                     uint32_t now_ms,
+                                                     uint32_t timeout_ms);
+MeterFrameEvent meter_frame_collector_feed(MeterFrameCollector &collector,
+                                            char byte,
+                                            uint32_t now_ms);
+
+enum class MeterParseStatus : uint8_t {
+  Valid = 0,
+  Empty,
+  MissingTerminator,
+  LineTooLong,
+  MissingRequired,
+  InvalidMeterSeconds,
+  MalformedValue
+};
+
+struct MeterParseResult {
+  float energy_total_kwh;
+  float total_power_w;
+  float phase_power_w[3];
+  uint32_t meter_seconds;
+  uint8_t fields_mask;
+  bool meter_seconds_valid;
+  bool valid;
+  MeterParseStatus status;
+};
+
+bool meter_parser_parse(const char *frame, size_t len, MeterParseResult &result);
+const char *meter_parse_status_text(MeterParseStatus status);
@@ -21,6 +21,13 @@ lib_deps =
 build_flags =
  -DSERIAL_DEBUG_MODE_FLAG=1

+[env:hil]
+extends = env:lilygo-t3-v1-6-1
+build_flags =
+  ${env:lilygo-t3-v1-6-1.build_flags}
+  -DENABLE_HIL_TRACE=1
+monitor_speed = 115200
+
 [env:lilygo-t3-v1-6-1-test]
 platform = https://github.com/pioarduino/platform-espressif32/releases/download/51.03.07/platform-espressif32.zip
 board = ttgo-lora32-v1
@@ -0,0 +1,4 @@
+[pytest]
+markers =
+    hil: validates captured hardware-in-the-loop evidence (never opens a serial device)
+testpaths = hil_tests
@@ -0,0 +1,2 @@
+pyserial>=3.5,<4
+pytest>=8,<10
@@ -0,0 +1,156 @@
+#include "hil_trace.h"
+
+#if ENABLE_HIL_TRACE
+
+#include <stdarg.h>
+#ifdef ARDUINO_ARCH_ESP32
+#include <esp_heap_caps.h>
+#endif
+
+static bool g_faults[static_cast<uint8_t>(HilFault::Count)] = {};
+static uint32_t g_delay_ack_ms = 1000;
+static uint32_t g_last_health_ms = 0;
+static String g_command;
+
+static const char *role_name(DeviceRole role) {
+  return role == DeviceRole::Sender ? "sender" : "receiver";
+}
+
+void hil_trace_event(const char *event, const char *fields_json) {
+  if (!event || !event[0]) {
+    return;
+  }
+  char line[448] = {};
+  int written = 0;
+  if (fields_json && fields_json[0]) {
+    written = snprintf(line, sizeof(line), "HIL:{\"event\":\"%s\",\"ms\":%lu,%s}\n",
+                       event, static_cast<unsigned long>(millis()), fields_json);
+  } else {
+    written = snprintf(line, sizeof(line), "HIL:{\"event\":\"%s\",\"ms\":%lu}\n",
+                       event, static_cast<unsigned long>(millis()));
+  }
+  if (written <= 0) return;
+  size_t length = static_cast<size_t>(written);
+  if (length >= sizeof(line)) length = sizeof(line) - 1;
+  Serial.write(reinterpret_cast<const uint8_t *>(line), length);
+}
+
+void hil_trace_eventf(const char *event, const char *fields_fmt, ...) {
+  char fields[320] = {};
+  if (fields_fmt && fields_fmt[0]) {
+    va_list args;
+    va_start(args, fields_fmt);
+    vsnprintf(fields, sizeof(fields), fields_fmt, args);
+    va_end(args);
+  }
+  hil_trace_event(event, fields);
+}
+
+void hil_trace_init(DeviceRole role, uint16_t short_id, int reset_reason) {
+  hil_trace_eventf("boot", "\"reset_reason\":%d,\"short_id\":%u", reset_reason, short_id);
+  hil_trace_eventf("role", "\"role\":\"%s\",\"short_id\":%u", role_name(role), short_id);
+}
+
+static bool arm_fault(const String &name) {
+  struct Mapping { const char *name; HilFault fault; };
+  static const Mapping mappings[] = {
+    {"drop_chunk", HilFault::DropChunk},
+    {"duplicate_chunk", HilFault::DuplicateChunk},
+    {"corrupt_chunk", HilFault::CorruptChunk},
+    {"suppress_ack", HilFault::SuppressAck},
+    {"delay_ack", HilFault::DelayAck},
+    {"wrong_ack_batch", HilFault::WrongAckBatch},
+  };
+  for (const Mapping &mapping : mappings) {
+    if (name == mapping.name) {
+      g_faults[static_cast<uint8_t>(mapping.fault)] = true;
+      hil_trace_eventf("fault_armed", "\"fault\":\"%s\"", mapping.name);
+      return true;
+    }
+  }
+  return false;
+}
+
+static void handle_command(const String &command) {
+  if (command == "clear") {
+    memset(g_faults, 0, sizeof(g_faults));
+    hil_trace_event("faults_cleared");
+    return;
+  }
+  if (command.startsWith("delay_ack:")) {
+    long requested = command.substring(10).toInt();
+    if (requested >= 0 && requested <= 10000) {
+      g_delay_ack_ms = static_cast<uint32_t>(requested);
+      arm_fault("delay_ack");
+      return;
+    }
+  }
+  if (!arm_fault(command)) {
+    hil_trace_eventf("command_rejected", "\"reason\":\"unknown\",\"length\":%u",
+                     static_cast<unsigned>(command.length()));
+  }
+}
+
+void hil_trace_poll_commands() {
+  // Commands are intentionally tiny and line-based to avoid dynamic JSON parsing
+  // or meaningful loop latency. Accepted form: HILCMD:<one-shot-fault>\n.
+  uint8_t budget = 64;
+  while (Serial.available() && budget-- > 0) {
+    char c = static_cast<char>(Serial.read());
+    if (c == '\r') {
+      continue;
+    }
+    if (c == '\n') {
+      if (g_command.startsWith("HILCMD:")) {
+        handle_command(g_command.substring(7));
+      }
+      g_command = "";
+    } else if (g_command.length() < 63) {
+      g_command += c;
+    } else {
+      g_command = "";
+    }
+  }
+}
+
+bool hil_fault_take(HilFault fault) {
+  uint8_t index = static_cast<uint8_t>(fault);
+  if (index >= static_cast<uint8_t>(HilFault::Count) || !g_faults[index]) {
+    return false;
+  }
+  g_faults[index] = false;
+  return true;
+}
+
+uint32_t hil_fault_delay_ms() {
+  return g_delay_ack_ms;
+}
+
+uint32_t hil_crc32(const uint8_t *data, size_t len) {
+  uint32_t crc = 0xFFFFFFFFUL;
+  for (size_t i = 0; i < len; ++i) {
+    crc ^= data[i];
+    for (uint8_t bit = 0; bit < 8; ++bit) {
+      crc = (crc >> 1) ^ ((crc & 1U) ? 0xEDB88320UL : 0);
+    }
+  }
+  return ~crc;
+}
+
+void hil_trace_health(uint8_t queue_depth, uint8_t build_depth, uint8_t retry_count) {
+  uint32_t now = millis();
+  if (now - g_last_health_ms < 5000) {
+    return;
+  }
+  g_last_health_ms = now;
+#ifdef ARDUINO_ARCH_ESP32
+  hil_trace_eventf("health", "\"heap_free\":%u,\"heap_min\":%u,\"stack_words\":%u,\"queue\":%u,\"build\":%u,\"retries\":%u",
+                   ESP.getFreeHeap(), ESP.getMinFreeHeap(), uxTaskGetStackHighWaterMark(nullptr),
+                   queue_depth, build_depth, retry_count);
+#else
+  hil_trace_eventf("health", "\"queue\":%u,\"build\":%u,\"retries\":%u",
+                   queue_depth, build_depth, retry_count);
+#endif
+}
+
+#endif
@@ -1,4 +1,5 @@
 #include "lora_transport.h"
+#include "hil_trace.h"
 #include <LoRa.h>
 #include <SPI.h>
 #include <math.h>
@@ -13,6 +14,7 @@ static void note_reject(RxRejectReason reason) {
    if (now_ms - g_last_rx_reject_log_ms >= 1000) {
      g_last_rx_reject_log_ms = now_ms;
      Serial.printf("lora_rx: reject reason=%s\n", rx_reject_reason_text(reason));
+      hil_trace_eventf("lora_reject", "\"reason\":\"%s\"", rx_reject_reason_text(reason));
    }
  }
 }
@@ -84,6 +86,13 @@ bool lora_send(const LoraPacket &pkt) {
  buffer[idx++] = static_cast<uint8_t>(crc >> 8);
  buffer[idx++] = static_cast<uint8_t>(crc & 0xFF);

+  if (pkt.payload_type == PayloadType::MeterBatch && hil_fault_take(HilFault::CorruptChunk) && idx > 7) {
+    // Flip a payload byte after calculating the frame CRC so the receiver's
+    // production CRC path rejects this one HIL frame.
+    buffer[5 + (pkt.payload_len > 6 ? 6 : 0)] ^= 0x01;
+    hil_trace_event("fault_injected", "\"fault\":\"corrupt_chunk\"");
+  }
+
  LoRa.beginPacket();
  if (SERIAL_DEBUG_MODE) {
    t2 = millis();
@@ -6,6 +6,7 @@
 #include "compressor.h"
 #include "lora_transport.h"
 #include "meter_driver.h"
+#include "meter_health.h"
 #include "power_manager.h"
 #include "time_manager.h"
 #include "wifi_manager.h"
@@ -14,11 +15,15 @@
 #include "display_ui.h"
 #include "test_mode.h"
 #include "sd_logger.h"
+#include "hil_trace.h"
 #include <stdarg.h>
 #include <math.h>
 #ifdef ARDUINO_ARCH_ESP32
 #include <esp_task_wdt.h>
 #include <esp_system.h>
+#include <freertos/FreeRTOS.h>
+#include <freertos/queue.h>
+#include <freertos/task.h>
 #endif

 static DeviceRole g_role = DeviceRole::Sender;
@@ -111,6 +116,17 @@ static MeterData g_last_meter_data = {};
 static bool g_last_meter_valid = false;
 static uint32_t g_last_meter_rx_ms = 0;
 static uint32_t g_meter_stale_seconds = 0;
+static MeterHealthState g_meter_health_state = {};
+static constexpr uint32_t METER_SAMPLE_MAX_AGE_MS = 15000;
+#ifdef ARDUINO_ARCH_ESP32
+struct MeterSampleEvent {
+  MeterData data;
+  uint32_t rx_ms;
+};
+static QueueHandle_t g_meter_sample_queue = nullptr;
+static TaskHandle_t g_meter_reader_task = nullptr;
+static bool g_meter_reader_task_running = false;
+#endif
 static constexpr uint32_t SENDER_TIMESYNC_ACQUIRE_MS = 10UL * 60UL * 1000UL;
 static constexpr uint32_t SENDER_TIMESYNC_ACQUIRE_INTERVAL_SEC = 20;
 static constexpr uint32_t SENDER_TIMESYNC_ACQUIRE_WINDOW_MS = 3000;
@@ -126,7 +142,73 @@ static void serial_debug_printf(const char *fmt, ...) {
  va_start(args, fmt);
  vsnprintf(buf, sizeof(buf), fmt, args);
  va_end(args);
-  Serial.println(buf);
+  Serial.printf("%s\n", buf);
+}
+
+static void set_last_meter_sample(const MeterData &parsed, uint32_t rx_ms) {
+  g_last_meter_data = parsed;
+  g_last_meter_valid = true;
+  g_last_meter_rx_ms = rx_ms;
+  g_meter_stale_seconds = 0;
+}
+
+static bool poll_and_parse_meter(MeterData &parsed) {
+  const char *frame = nullptr;
+  size_t frame_len = 0;
+  if (!meter_poll_frame(frame, frame_len)) return false;
+  parsed = {};
+  parsed.energy_total_kwh = NAN;
+  parsed.total_power_w = NAN;
+  parsed.phase_power_w[0] = NAN;
+  parsed.phase_power_w[1] = NAN;
+  parsed.phase_power_w[2] = NAN;
+  parsed.valid = false;
+  return meter_parse_frame(frame, frame_len, parsed);
+}
+
+#ifdef ARDUINO_ARCH_ESP32
+static void meter_reader_task_entry(void *) {
+  for (;;) {
+    MeterData parsed = {};
+    if (!poll_and_parse_meter(parsed)) {
+      vTaskDelay(pdMS_TO_TICKS(5));
+      continue;
+    }
+    MeterSampleEvent event = {};
+    event.data = parsed;
+    event.rx_ms = millis();
+    xQueueOverwrite(g_meter_sample_queue, &event);
+  }
+}
+
+static bool meter_reader_start() {
+  g_meter_sample_queue = xQueueCreate(1, sizeof(MeterSampleEvent));
+  if (!g_meter_sample_queue) return false;
+  BaseType_t rc = xTaskCreatePinnedToCore(meter_reader_task_entry, "meter_reader", 4096,
+                                          nullptr, 2, &g_meter_reader_task, 0);
+  if (rc != pdPASS) {
+    vQueueDelete(g_meter_sample_queue);
+    g_meter_sample_queue = nullptr;
+    return false;
+  }
+  g_meter_reader_task_running = true;
+  serial_debug_printf("meter: reader task started rx_buffer=8192");
+  return true;
+}
+#endif
+
+static void meter_reader_pump(uint32_t now_ms) {
+#ifdef ARDUINO_ARCH_ESP32
+  if (g_meter_reader_task_running && g_meter_sample_queue) {
+    MeterSampleEvent event = {};
+    if (xQueueReceive(g_meter_sample_queue, &event, 0) == pdTRUE) {
+      set_last_meter_sample(event.data, event.rx_ms);
+    }
+    return;
+  }
+#endif
+  MeterData parsed = {};
+  if (poll_and_parse_meter(parsed)) set_last_meter_sample(parsed, now_ms);
 }

 static void sender_set_timesync_mode(uint8_t mode) {
@@ -216,7 +298,7 @@ static bool sender_timesync_window_due() {
  return false;
 }

-static bool batch_queue_drop_oldest() {
+static bool batch_queue_remove_oldest(bool overflow_drop) {
  if (g_batch_count == 0) {
    return false;
  }
@@ -231,6 +313,12 @@ static bool batch_queue_drop_oldest() {
  }
  g_batch_tail = (g_batch_tail + 1) % BATCH_QUEUE_DEPTH;
  g_batch_count--;
+  if (overflow_drop) {
+    hil_trace_eventf("queue_drop", "\"queue\":%u,\"inflight\":%s", g_batch_count,
+                     dropped_inflight ? "true" : "false");
+  } else {
+    hil_trace_eventf("queue", "\"action\":\"dequeue_acked\",\"queue\":%u", g_batch_count);
+  }
  return dropped_inflight;
 }

@@ -279,7 +367,7 @@ static void batch_queue_enqueue(const MeterData *samples, uint8_t count) {
    return;
  }
  if (g_batch_count >= BATCH_QUEUE_DEPTH) {
-    if (batch_queue_drop_oldest()) {
+    if (batch_queue_remove_oldest(true)) {
      g_batch_id++;
    }
  }
@@ -292,6 +380,8 @@ static void batch_queue_enqueue(const MeterData *samples, uint8_t count) {
  }
  g_batch_head = (g_batch_head + 1) % BATCH_QUEUE_DEPTH;
  g_batch_count++;
+  hil_trace_eventf("queue", "\"action\":\"enqueue\",\"queue\":%u,\"samples\":%u",
+                   g_batch_count, count);
 }

 static uint32_t last_sample_ts() {
@@ -482,8 +572,23 @@ static bool send_batch_payload(const uint8_t *data, size_t len, uint32_t ts_for_

    watchdog_kick();
    uint32_t tx_start = millis();
-    bool ok = lora_send(pkt);
+    bool dropped = hil_fault_take(HilFault::DropChunk);
+    bool duplicate = false;
+    bool ok = dropped ? true : lora_send(pkt);
+    if (!dropped && hil_fault_take(HilFault::DuplicateChunk)) {
+      duplicate = true;
+      delay(10);
+      ok = lora_send(pkt) && ok;
+    }
    uint32_t tx_ms = millis() - tx_start;
+    hil_trace_eventf("chunk_tx", "\"batch_id\":%u,\"index\":%u,\"count\":%u,\"len\":%u,\"crc32\":%lu,\"ok\":%s,\"dropped\":%s,\"duplicate\":%s",
+                     batch_id, i, chunk_count, static_cast<unsigned>(chunk_len),
+                     static_cast<unsigned long>(hil_crc32(data + offset, chunk_len)), ok ? "true" : "false",
+                     dropped ? "true" : "false", duplicate ? "true" : "false");
+    if (dropped || duplicate) {
+      hil_trace_eventf("fault_injected", "\"fault\":\"%s\",\"batch_id\":%u,\"index\":%u",
+                       dropped ? "drop_chunk" : "duplicate_chunk", batch_id, i);
+    }
    all_ok = all_ok && ok;
    if (!ok) {
      note_fault(g_sender_faults, g_sender_last_error, g_sender_last_error_utc, g_sender_last_error_ms, FaultType::LoraTx);
@@ -502,26 +607,47 @@ static bool send_batch_payload(const uint8_t *data, size_t len, uint32_t ts_for_
 }

 static void send_batch_ack(uint16_t batch_id, uint16_t sender_id) {
+  if (hil_fault_take(HilFault::SuppressAck)) {
+    hil_trace_eventf("fault_injected", "\"fault\":\"suppress_ack\",\"batch_id\":%u", batch_id);
+    hil_trace_eventf("ack_tx", "\"batch_id\":%u,\"sender\":%u,\"ok\":false,\"suppressed\":true", batch_id, sender_id);
+    lora_receive_continuous();
+    return;
+  }
+  if (hil_fault_take(HilFault::DelayAck)) {
+    uint32_t delay_ms = hil_fault_delay_ms();
+    hil_trace_eventf("fault_injected", "\"fault\":\"delay_ack\",\"batch_id\":%u,\"delay_ms\":%lu",
+                     batch_id, static_cast<unsigned long>(delay_ms));
+    delay(delay_ms);
+  }
  LoraPacket ack = {};
  ack.protocol_version = PROTOCOL_VERSION;
  ack.role = DeviceRole::Receiver;
  ack.device_id_short = g_short_id;
  ack.payload_type = PayloadType::Ack;
  ack.payload_len = 6;
-  write_u16_le(&ack.payload[0], batch_id);
+  uint16_t transmitted_batch_id = batch_id;
+  if (hil_fault_take(HilFault::WrongAckBatch)) {
+    transmitted_batch_id = static_cast<uint16_t>(batch_id + 1);
+    hil_trace_eventf("fault_injected", "\"fault\":\"wrong_ack_batch\",\"batch_id\":%u,\"sent_batch_id\":%u",
+                     batch_id, transmitted_batch_id);
+  }
+  write_u16_le(&ack.payload[0], transmitted_batch_id);
  write_u16_le(&ack.payload[2], sender_id);
  write_u16_le(&ack.payload[4], g_short_id);
  uint8_t repeats = ACK_REPEAT_COUNT == 0 ? 1 : ACK_REPEAT_COUNT;
+  bool all_ok = true;
  for (uint8_t i = 0; i < repeats; ++i) {
    if (SERIAL_DEBUG_MODE) {
      serial_debug_printf("ack: tx repeat %u/%u batch_id=%u", static_cast<unsigned>(i + 1),
                          static_cast<unsigned>(repeats), batch_id);
    }
-    lora_send(ack);
+    all_ok = lora_send(ack) && all_ok;
    if (i + 1 < repeats && ACK_REPEAT_DELAY_MS > 0) {
      delay(ACK_REPEAT_DELAY_MS);
    }
  }
+  hil_trace_eventf("ack_tx", "\"batch_id\":%u,\"sent_batch_id\":%u,\"sender\":%u,\"repeats\":%u,\"ok\":%s",
+                   batch_id, transmitted_batch_id, sender_id, repeats, all_ok ? "true" : "false");
  lora_receive_continuous();
 }

@@ -543,6 +669,8 @@ static bool prepare_inflight_from_queue() {
    g_inflight_samples[i] = batch->samples[i];
  }
  g_inflight_active = true;
+  hil_trace_eventf("batch_created", "\"batch_id\":%u,\"samples\":%u,\"queue\":%u",
+                   g_inflight_batch_id, g_inflight_count, g_batch_count);
  return true;
 }

@@ -576,8 +704,13 @@ static bool send_inflight_batch(uint32_t ts_for_display) {
  size_t encoded_len = 0;
  uint32_t encode_start = millis();
  if (!encode_batch(input, encoded, sizeof(encoded), &encoded_len)) {
+    hil_trace_eventf("payload_encode", "\"batch_id\":%u,\"ok\":false,\"samples\":%u",
+                     g_inflight_batch_id, g_inflight_count);
    return false;
  }
+  hil_trace_eventf("payload_encode", "\"batch_id\":%u,\"ok\":true,\"samples\":%u,\"len\":%u,\"crc32\":%lu",
+                   g_inflight_batch_id, g_inflight_count, static_cast<unsigned>(encoded_len),
+                   static_cast<unsigned long>(hil_crc32(encoded, encoded_len)));
  uint32_t encode_ms = millis() - encode_start;
  if (SERIAL_DEBUG_MODE) {
    serial_debug_printf("tx: batch_id=%u count=%u bin_len=%u", g_inflight_batch_id, g_inflight_count,
@@ -628,7 +761,7 @@ static bool resend_inflight_batch(uint32_t ts_for_display) {

 static void finish_inflight_batch() {
  if (g_batch_count > 0) {
-    batch_queue_drop_oldest();
+    batch_queue_remove_oldest(false);
  }
  g_batch_ack_pending = false;
  g_batch_retry_count = 0;
@@ -661,9 +794,21 @@ static bool process_batch_packet(const LoraPacket &pkt, BatchInput &out_batch, b
  const uint8_t *chunk_data = &pkt.payload[BATCH_HEADER_SIZE];
  size_t chunk_len = pkt.payload_len - BATCH_HEADER_SIZE;
  uint32_t now_ms = millis();
+  hil_trace_eventf("chunk_rx", "\"batch_id\":%u,\"index\":%u,\"count\":%u,\"len\":%u,\"crc32\":%lu",
+                   batch_id, chunk_index, chunk_count, static_cast<unsigned>(chunk_len),
+                   static_cast<unsigned long>(hil_crc32(chunk_data, chunk_len)));

-  if (!g_batch_rx.active || batch_id != g_batch_rx.batch_id || (now_ms - g_batch_rx.last_rx_ms > g_batch_rx.timeout_ms)) {
+  bool rx_timed_out = g_batch_rx.active && (now_ms - g_batch_rx.last_rx_ms > g_batch_rx.timeout_ms);
+  bool batch_changed = g_batch_rx.active && batch_id != g_batch_rx.batch_id;
+  if (rx_timed_out || batch_changed) {
+    hil_trace_eventf("reassembly_incomplete", "\"batch_id\":%u,\"received\":%u,\"expected_chunks\":%u,\"reason\":\"%s\"",
+                     g_batch_rx.batch_id, g_batch_rx.received_len, g_batch_rx.expected_chunks,
+                     rx_timed_out ? "timeout" : "batch_changed");
+  }
+  if (!g_batch_rx.active || batch_changed || rx_timed_out) {
    if (chunk_index != 0) {
+      hil_trace_eventf("reassembly_incomplete", "\"batch_id\":%u,\"index\":%u,\"reason\":\"missing_first_chunk\"",
+                       batch_id, chunk_index);
      reset_batch_rx();
      return false;
    }
@@ -681,11 +826,14 @@ static bool process_batch_packet(const LoraPacket &pkt, BatchInput &out_batch, b
  }

  if (!g_batch_rx.active || chunk_index != g_batch_rx.next_index || chunk_count != g_batch_rx.expected_chunks) {
+    hil_trace_eventf("reassembly_incomplete", "\"batch_id\":%u,\"index\":%u,\"expected_index\":%u,\"reason\":\"sequence\"",
+                     batch_id, chunk_index, g_batch_rx.next_index);
    reset_batch_rx();
    return false;
  }

  if (g_batch_rx.received_len + chunk_len > g_batch_rx.total_len || g_batch_rx.received_len + chunk_len > BATCH_MAX_COMPRESSED) {
+    hil_trace_eventf("reassembly_incomplete", "\"batch_id\":%u,\"reason\":\"length\"", batch_id);
    reset_batch_rx();
    return false;
  }
@@ -698,10 +846,16 @@ static bool process_batch_packet(const LoraPacket &pkt, BatchInput &out_batch, b
  if (g_batch_rx.next_index == g_batch_rx.expected_chunks && g_batch_rx.received_len == g_batch_rx.total_len) {
    if (!decode_batch(g_batch_rx.buffer, g_batch_rx.received_len, &out_batch)) {
      decode_error = true;
+      hil_trace_eventf("payload_decode", "\"batch_id\":%u,\"ok\":false,\"len\":%u",
+                       batch_id, g_batch_rx.received_len);
      reset_batch_rx();
      return false;
    }
    out_batch_id = batch_id;
+    hil_trace_eventf("reassembly_complete", "\"batch_id\":%u,\"chunks\":%u,\"len\":%u,\"crc32\":%lu",
+                     batch_id, g_batch_rx.expected_chunks, g_batch_rx.received_len,
+                     static_cast<unsigned long>(hil_crc32(g_batch_rx.buffer, g_batch_rx.received_len)));
+    hil_trace_eventf("payload_decode", "\"batch_id\":%u,\"ok\":true,\"samples\":%u", batch_id, out_batch.n);
    reset_batch_rx();
    return true;
  }
@@ -720,6 +874,11 @@ void setup() {
  g_boot_ms = millis();
  g_role = detect_role();
  init_device_ids(g_short_id, g_device_id, sizeof(g_device_id));
+  int reset_reason = 0;
+#ifdef ARDUINO_ARCH_ESP32
+  reset_reason = static_cast<int>(esp_reset_reason());
+#endif
+  hil_trace_init(g_role, g_short_id, reset_reason);
  display_set_role(g_role);
  if (SERIAL_DEBUG_MODE) {
 #ifdef ARDUINO_ARCH_ESP32
@@ -739,6 +898,11 @@ void setup() {
    power_sender_init();
    power_configure_unused_pins_sender();
    meter_init();
+#ifdef ARDUINO_ARCH_ESP32
+    if (!meter_reader_start()) {
+      serial_debug_printf("meter: reader task unavailable, using inline polling");
+    }
+#endif
    g_last_sample_ms = millis() - METER_SAMPLE_INTERVAL_MS;
    g_last_send_ms = millis();
    update_battery_cache();
@@ -779,7 +943,9 @@ void setup() {

 static void sender_loop() {
  watchdog_kick();
+  hil_trace_poll_commands();
  uint32_t now_ms = millis();
+  hil_trace_health(g_batch_count, g_build_count, g_batch_retry_count);
  display_set_sender_queue(g_batch_count, g_build_count > 0);
  display_set_sender_batches(g_last_acked_batch_id, g_batch_id);
  if (SERIAL_DEBUG_MODE && now_ms - g_last_debug_log_ms >= 5000) {
@@ -794,23 +960,7 @@ static void sender_loop() {
                        g_batch_retry_count);
  }

-  const char *frame = nullptr;
-  size_t frame_len = 0;
-  if (meter_poll_frame(frame, frame_len)) {
-    MeterData parsed = {};
-    parsed.energy_total_kwh = NAN;
-    parsed.total_power_w = NAN;
-    parsed.phase_power_w[0] = NAN;
-    parsed.phase_power_w[1] = NAN;
-    parsed.phase_power_w[2] = NAN;
-    parsed.valid = false;
-    if (meter_parse_frame(frame, frame_len, parsed)) {
-      g_last_meter_data = parsed;
-      g_last_meter_valid = true;
-      g_last_meter_rx_ms = now_ms;
-      g_meter_stale_seconds = 0;
-    }
-  }
+  meter_reader_pump(now_ms);

  if (now_ms - g_last_sample_ms >= METER_SAMPLE_INTERVAL_MS) {
    g_last_sample_ms = now_ms;
@@ -818,21 +968,28 @@ static void sender_loop() {
    data.short_id = g_short_id;
    strncpy(data.device_id, g_device_id, sizeof(data.device_id));

-    bool meter_ok = g_last_meter_valid;
-    if (meter_ok) {
+    uint32_t meter_age_ms = g_last_meter_valid ? now_ms - g_last_meter_rx_ms : UINT32_MAX;
+    MeterHealthEvaluation meter_health = meter_health_evaluate(
+        g_meter_health_state, g_last_meter_valid, meter_age_ms, METER_SAMPLE_MAX_AGE_MS);
+    bool meter_ok = meter_health.ok;
+    if (g_last_meter_valid) {
      data.energy_total_kwh = g_last_meter_data.energy_total_kwh;
      data.total_power_w = g_last_meter_data.total_power_w;
      data.phase_power_w[0] = g_last_meter_data.phase_power_w[0];
      data.phase_power_w[1] = g_last_meter_data.phase_power_w[1];
      data.phase_power_w[2] = g_last_meter_data.phase_power_w[2];
-      uint32_t age_ms = now_ms - g_last_meter_rx_ms;
-      g_meter_stale_seconds = age_ms >= 1000 ? (age_ms / 1000) : 0;
+      g_meter_stale_seconds = meter_age_ms >= 1000 ? (meter_age_ms / 1000) : 0;
    } else {
      g_meter_stale_seconds++;
    }
-    if (!meter_ok) {
+    if (meter_health.fault_started) {
      note_fault(g_sender_faults, g_sender_last_error, g_sender_last_error_utc, g_sender_last_error_ms, FaultType::MeterRead);
      display_set_last_error(g_sender_last_error, g_sender_last_error_utc, g_sender_last_error_ms);
+      hil_trace_eventf("meter_fault", "\"state\":\"active\",\"classification\":\"%s\",\"stale_s\":%lu",
+                       meter_health_classification_text(meter_health.classification),
+                       static_cast<unsigned long>(g_meter_stale_seconds));
+    } else if (meter_health.recovered) {
+      hil_trace_event("meter_fault", "\"state\":\"recovered\"");
    }
    if (g_build_count == 0 && battery_sample_due(now_ms)) {
      update_battery_cache();
@@ -844,12 +1001,19 @@ static void sender_loop() {
    uint32_t now_utc = time_get_utc();
    data.ts_utc = now_utc > 0 ? now_utc : millis() / 1000;
    data.valid = meter_ok;
+    hil_trace_eventf("meter_sample", "\"ok\":%s,\"classification\":\"%s\",\"ts\":%lu,\"stale_s\":%lu",
+                     meter_ok ? "true" : "false",
+                     meter_health_classification_text(meter_health.classification),
+                     static_cast<unsigned long>(data.ts_utc),
+                     static_cast<unsigned long>(g_meter_stale_seconds));

-    g_last_sample_ts_utc = data.ts_utc;
-    g_build_samples[g_build_count++] = data;
-    if (g_build_count >= METER_BATCH_MAX_SAMPLES) {
-      batch_queue_enqueue(g_build_samples, g_build_count);
-      g_build_count = 0;
+    if (meter_ok) {
+      g_last_sample_ts_utc = data.ts_utc;
+      g_build_samples[g_build_count++] = data;
+      if (g_build_count >= METER_BATCH_MAX_SAMPLES) {
+        batch_queue_enqueue(g_build_samples, g_build_count);
+        g_build_count = 0;
+      }
    }
    display_set_last_meter(data);
    display_set_last_read(meter_ok, data.ts_utc);
@@ -899,6 +1063,7 @@ static void sender_loop() {
      if (ack_sender == g_short_id && ack_receiver == ack_pkt.device_id_short &&
          g_batch_ack_pending && ack_id == g_last_sent_batch_id) {
        g_last_acked_batch_id = ack_id;
+        hil_trace_eventf("ack_rx", "\"batch_id\":%u,\"ok\":true,\"receiver\":%u", ack_id, ack_receiver);
        serial_debug_printf("ack: rx ok batch_id=%u", ack_id);
        finish_inflight_batch();
      } else {
@@ -907,6 +1072,8 @@ static void sender_loop() {
        } else if (ack_id != g_last_sent_batch_id) {
          sender_note_rx_reject(RxRejectReason::BatchIdMismatch, "ack");
        }
+        hil_trace_eventf("ack_rx", "\"batch_id\":%u,\"ok\":false,\"expected_batch_id\":%u,\"sender\":%u,\"receiver\":%u",
+                         ack_id, g_last_sent_batch_id, ack_sender, ack_receiver);
        if (SERIAL_DEBUG_MODE) {
          serial_debug_printf("ack: reject batch_id=%u sender=%u receiver=%u exp_batch=%u exp_sender=%u",
                              ack_id, ack_sender, ack_receiver, g_last_sent_batch_id, g_short_id);
@@ -917,6 +1084,8 @@ static void sender_loop() {

  bool timesync_due = (!g_batch_ack_pending && sender_timesync_window_due());
  if (timesync_due) {
+    hil_trace_eventf("time_bootstrap", "\"stage\":\"request\",\"window_ms\":%lu",
+                     static_cast<unsigned long>((g_sender_timesync_mode == 2) ? SENDER_TIMESYNC_ACQUIRE_WINDOW_MS : SENDER_TIMESYNC_WINDOW_MS));
    LoraPacket rx = {};
    uint32_t rx_start = millis();
    uint32_t window_ms = (g_sender_timesync_mode == 2) ? SENDER_TIMESYNC_ACQUIRE_WINDOW_MS : SENDER_TIMESYNC_WINDOW_MS;
@@ -941,6 +1110,8 @@ static void sender_loop() {
        display_set_last_error(FaultType::None, 0, 0);
      }
      serial_debug_printf("timesync: rx ok window_ms=%lu", static_cast<unsigned long>(window_ms));
+      hil_trace_eventf("time_bootstrap", "\"stage\":\"complete\",\"ok\":true,\"utc\":%lu,\"receiver\":%u",
+                       static_cast<unsigned long>(time_get_utc()), rx.device_id_short);
    } else {
      sender_note_rx_reject(RxRejectReason::LengthMismatch, "timesync");
      if (SERIAL_DEBUG_MODE) {
@@ -961,9 +1132,13 @@ static void sender_loop() {
  if (g_batch_ack_pending && (now_ms - g_last_batch_send_ms >= g_batch_ack_timeout_ms)) {
    if (g_batch_retry_count < BATCH_MAX_RETRIES) {
      g_batch_retry_count++;
+      hil_trace_eventf("retry", "\"batch_id\":%u,\"retry\":%u,\"reason\":\"ack_timeout\"",
+                       g_inflight_batch_id, g_batch_retry_count);
      serial_debug_printf("ack: timeout batch_id=%u retry=%u", g_inflight_batch_id, g_batch_retry_count);
      resend_inflight_batch(last_sample_ts());
    } else {
+      hil_trace_eventf("timeout", "\"batch_id\":%u,\"retries\":%u,\"kind\":\"ack\"",
+                       g_inflight_batch_id, g_batch_retry_count);
      serial_debug_printf("ack: failed batch_id=%u policy=%s", g_inflight_batch_id,
                          BATCH_RETRY_POLICY == BatchRetryPolicy::Drop ? "drop" : "keep");
      if (BATCH_RETRY_POLICY == BatchRetryPolicy::Drop) {
@@ -1002,6 +1177,8 @@ static void sender_loop() {

 static void receiver_loop() {
  watchdog_kick();
+  hil_trace_poll_commands();
+  hil_trace_health(0, 0, 0);
  if (g_last_timesync_ms == 0) {
    g_last_timesync_ms = millis() - (TIME_SYNC_INTERVAL_SEC * 1000UL - TIME_SYNC_OFFSET_MS);
  }
@@ -1069,6 +1246,7 @@ static void receiver_loop() {
        }
        bool duplicate = sender_idx >= 0 && g_last_batch_id_rx[sender_idx] == batch_id;
        if (duplicate) {
+          hil_trace_eventf("duplicate", "\"batch_id\":%u,\"sender\":%u", batch_id, pkt.device_id_short);
          send_batch_ack(batch_id, pkt.device_id_short);
        } else {
          g_last_batch_id_rx[sender_idx] = batch_id;
@@ -1172,23 +1350,29 @@ static void receiver_loop() {
      }
    }
    if (burst_sent) {
-      if (!time_send_timesync(g_short_id)) {
+      bool time_sent = time_send_timesync(g_short_id);
+      if (!time_sent) {
        note_fault(g_receiver_faults, g_receiver_last_error, g_receiver_last_error_utc, g_receiver_last_error_ms, FaultType::LoraTx);
        display_set_last_error(g_receiver_last_error, g_receiver_last_error_utc, g_receiver_last_error_ms);
      }
      if (SERIAL_DEBUG_MODE) {
        serial_debug_printf("timesync: tx burst");
      }
+      hil_trace_eventf("time_bootstrap", "\"stage\":\"transmit\",\"mode\":\"burst\",\"ok\":%s,\"utc\":%lu",
+                       time_sent ? "true" : "false", static_cast<unsigned long>(time_get_utc()));
      g_last_timesync_ms = now_ms;
    } else if (now_ms - g_last_timesync_ms > interval_sec * 1000UL) {
      g_last_timesync_ms = now_ms;
-      if (!time_send_timesync(g_short_id)) {
+      bool time_sent = time_send_timesync(g_short_id);
+      if (!time_sent) {
        note_fault(g_receiver_faults, g_receiver_last_error, g_receiver_last_error_utc, g_receiver_last_error_ms, FaultType::LoraTx);
        display_set_last_error(g_receiver_last_error, g_receiver_last_error_utc, g_receiver_last_error_ms);
      }
      if (SERIAL_DEBUG_MODE) {
        serial_debug_printf("timesync: tx normal");
      }
+      hil_trace_eventf("time_bootstrap", "\"stage\":\"transmit\",\"mode\":\"normal\",\"ok\":%s,\"utc\":%lu",
+                       time_sent ? "true" : "false", static_cast<unsigned long>(time_get_utc()));
    }
  }

@@ -1,155 +1,105 @@
 #include "meter_driver.h"
 #include "config.h"
+#include "hil_trace.h"
+#include "meter_parser.h"
+
 #include <math.h>
-#include <stdlib.h>
-#include <string.h>

 static constexpr uint32_t METER_FRAME_TIMEOUT_MS = 1500;
-static constexpr size_t METER_FRAME_MAX = 512;
+static constexpr size_t METER_UART_RX_BUFFER_BYTES = 8192;

-enum class MeterRxState : uint8_t {
-  WaitStart = 0,
-  InFrame = 1
-};
-
-static MeterRxState g_rx_state = MeterRxState::WaitStart;
-static char g_frame_buf[METER_FRAME_MAX + 1];
-static size_t g_frame_len = 0;
-static uint32_t g_last_rx_ms = 0;
+static MeterFrameCollector g_collector = {};
 static uint32_t g_bytes_rx = 0;
 static uint32_t g_frames_ok = 0;
 static uint32_t g_frames_parse_fail = 0;
 static uint32_t g_rx_overflow = 0;
 static uint32_t g_rx_timeout = 0;
+static volatile uint32_t g_uart_buffer_full = 0;
+static volatile uint32_t g_uart_fifo_overflow = 0;
+static volatile uint32_t g_uart_frame_error = 0;
+static volatile uint32_t g_uart_parity_error = 0;
 static uint32_t g_last_log_ms = 0;
+static uint32_t g_last_good_frame_ms = 0;
+#if ENABLE_HIL_TRACE
+static bool g_meter_seconds_seen = false;
+static uint32_t g_last_meter_seconds = 0;
+#endif

 void meter_init() {
+  meter_frame_collector_init(g_collector);
+#ifdef ARDUINO_ARCH_ESP32
+  Serial2.setRxBufferSize(METER_UART_RX_BUFFER_BYTES);
+#endif
  Serial2.begin(9600, SERIAL_7E1, PIN_METER_RX, -1);
-}
-
-static bool parse_obis_ascii_value(const char *line, const char *obis, float &out_value) {
-  const char *p = strstr(line, obis);
-  if (!p) {
-    return false;
-  }
-  const char *lparen = strchr(p, '(');
-  if (!lparen) {
-    return false;
-  }
-  const char *cur = lparen + 1;
-  char num_buf[24];
-  size_t n = 0;
-  while (*cur && *cur != ')' && *cur != '*') {
-    char c = *cur++;
-    if ((c >= '0' && c <= '9') || c == '-' || c == '+' || c == '.' || c == ',') {
-      if (c == ',') {
-        c = '.';
-      }
-      if (n + 1 < sizeof(num_buf)) {
-        num_buf[n++] = c;
-      }
-    } else if (n == 0) {
-      continue;
-    } else {
-      break;
+#ifdef ARDUINO_ARCH_ESP32
+  Serial2.onReceiveError([](hardwareSerial_error_t error) {
+    if (error == UART_BUFFER_FULL_ERROR) {
+      g_uart_buffer_full = g_uart_buffer_full + 1;
+    } else if (error == UART_FIFO_OVF_ERROR) {
+      g_uart_fifo_overflow = g_uart_fifo_overflow + 1;
+    } else if (error == UART_FRAME_ERROR) {
+      g_uart_frame_error = g_uart_frame_error + 1;
+    } else if (error == UART_PARITY_ERROR) {
+      g_uart_parity_error = g_uart_parity_error + 1;
    }
-  }
-  if (n == 0) {
-    return false;
-  }
-  num_buf[n] = '\0';
-  out_value = static_cast<float>(atof(num_buf));
-  return true;
-}
-
-static bool parse_obis_ascii_unit_scale(const char *line, const char *obis, float &value) {
-  const char *p = strstr(line, obis);
-  if (!p) {
-    return false;
-  }
-  const char *asterisk = strchr(p, '*');
-  if (!asterisk) {
-    return false;
-  }
-  const char *end = strchr(asterisk, ')');
-  if (!end) {
-    return false;
-  }
-  char unit_buf[8];
-  size_t ulen = 0;
-  for (const char *c = asterisk + 1; c < end && ulen + 1 < sizeof(unit_buf); ++c) {
-    if (*c == ' ') {
-      continue;
-    }
-    unit_buf[ulen++] = *c;
-  }
-  unit_buf[ulen] = '\0';
-  if (ulen == 0) {
-    return false;
-  }
-  if (strcmp(unit_buf, "Wh") == 0) {
-    value *= 0.001f;
-    return true;
-  }
-  return false;
+  });
+#endif
 }

 static void meter_debug_log() {
-  if (!SERIAL_DEBUG_MODE) {
-    return;
-  }
+  if (!SERIAL_DEBUG_MODE) return;
  uint32_t now_ms = millis();
-  if (now_ms - g_last_log_ms < 60000) {
-    return;
-  }
+  if (now_ms - g_last_log_ms < 60000) return;
  g_last_log_ms = now_ms;
-  Serial.printf("meter: ok=%lu parse_fail=%lu overflow=%lu timeout=%lu bytes=%lu\n",
+  Serial.printf("meter: ok=%lu parse_fail=%lu frame_ovf=%lu timeout=%lu bytes=%lu uart_full=%lu uart_fifo_ovf=%lu uart_frame=%lu uart_parity=%lu\n",
                static_cast<unsigned long>(g_frames_ok),
                static_cast<unsigned long>(g_frames_parse_fail),
                static_cast<unsigned long>(g_rx_overflow),
                static_cast<unsigned long>(g_rx_timeout),
-                static_cast<unsigned long>(g_bytes_rx));
+                static_cast<unsigned long>(g_bytes_rx),
+                static_cast<unsigned long>(g_uart_buffer_full),
+                static_cast<unsigned long>(g_uart_fifo_overflow),
+                static_cast<unsigned long>(g_uart_frame_error),
+                static_cast<unsigned long>(g_uart_parity_error));
+}
+
+void meter_get_stats(MeterDriverStats &out) {
+  out.frames_ok = g_frames_ok;
+  out.frames_parse_fail = g_frames_parse_fail;
+  out.rx_overflow = g_rx_overflow;
+  out.rx_timeout = g_rx_timeout;
+  out.bytes_rx = g_bytes_rx;
+  out.last_rx_ms = g_collector.last_rx_ms;
+  out.last_good_frame_ms = g_last_good_frame_ms;
+  out.uart_buffer_full = g_uart_buffer_full;
+  out.uart_fifo_overflow = g_uart_fifo_overflow;
+  out.uart_frame_error = g_uart_frame_error;
+  out.uart_parity_error = g_uart_parity_error;
 }

 bool meter_poll_frame(const char *&frame, size_t &len) {
  frame = nullptr;
  len = 0;
  uint32_t now_ms = millis();
-
-  if (g_rx_state == MeterRxState::InFrame && (now_ms - g_last_rx_ms > METER_FRAME_TIMEOUT_MS)) {
+  MeterFrameEvent timeout_event = meter_frame_collector_check_timeout(
+      g_collector, now_ms, METER_FRAME_TIMEOUT_MS);
+  if (timeout_event == MeterFrameEvent::Timeout) {
    g_rx_timeout++;
-    g_rx_state = MeterRxState::WaitStart;
-    g_frame_len = 0;
+    hil_trace_event("meter_frame", "\"ok\":false,\"classification\":\"timeout\"");
  }

  while (Serial2.available()) {
-    char c = static_cast<char>(Serial2.read());
+    char byte = static_cast<char>(Serial2.read());
    g_bytes_rx++;
-    g_last_rx_ms = now_ms;
-
-    if (g_rx_state == MeterRxState::WaitStart) {
-      if (c == '/') {
-        g_rx_state = MeterRxState::InFrame;
-        g_frame_len = 0;
-        g_frame_buf[g_frame_len++] = c;
-      }
-      continue;
-    }
-
-    if (g_frame_len + 1 >= sizeof(g_frame_buf)) {
+    MeterFrameEvent event = meter_frame_collector_feed(g_collector, byte, millis());
+    if (event == MeterFrameEvent::Resync) {
+      hil_trace_event("meter_resync", "\"reason\":\"new_start\"");
+    } else if (event == MeterFrameEvent::Overflow) {
      g_rx_overflow++;
-      g_rx_state = MeterRxState::WaitStart;
-      g_frame_len = 0;
-      continue;
-    }
-
-    g_frame_buf[g_frame_len++] = c;
-    if (c == '!') {
-      g_frame_buf[g_frame_len] = '\0';
-      frame = g_frame_buf;
-      len = g_frame_len;
-      g_rx_state = MeterRxState::WaitStart;
-      g_frame_len = 0;
+      hil_trace_event("meter_frame", "\"ok\":false,\"classification\":\"oversized\"");
+    } else if (event == MeterFrameEvent::Complete) {
+      frame = g_collector.buffer;
+      len = g_collector.length;
      meter_debug_log();
      return true;
    }
@@ -160,91 +110,43 @@ bool meter_poll_frame(const char *&frame, size_t &len) {
 }

 bool meter_parse_frame(const char *frame, size_t len, MeterData &data) {
-  if (!frame || len == 0) {
-    return false;
-  }
-  bool got_any = false;
-  bool energy_ok = false;
-  bool total_p_ok = false;
-  bool p1_ok = false;
-  bool p2_ok = false;
-  bool p3_ok = false;
-  char line[128];
-  size_t line_len = 0;
-
-  for (size_t i = 0; i < len; ++i) {
-    char c = frame[i];
-    if (c == '\r') {
-      continue;
-    }
-    if (c == '!') {
-      if (line_len + 1 < sizeof(line)) {
-        line[line_len++] = c;
-      }
-      line[line_len] = '\0';
-      data.valid = energy_ok || total_p_ok || p1_ok || p2_ok || p3_ok;
-      if (data.valid) {
-        g_frames_ok++;
-      } else {
-        g_frames_parse_fail++;
-      }
-      return data.valid;
-    }
-    if (c == '\n') {
-      line[line_len] = '\0';
-      if (line[0] == '!') {
-        data.valid = energy_ok || total_p_ok || p1_ok || p2_ok || p3_ok;
-        if (data.valid) {
-          g_frames_ok++;
-        } else {
-          g_frames_parse_fail++;
-        }
-        return data.valid;
-      }
-
-      float value = NAN;
-      if (parse_obis_ascii_value(line, "1-0:1.8.0", value)) {
-        parse_obis_ascii_unit_scale(line, "1-0:1.8.0", value);
-        data.energy_total_kwh = value;
-        energy_ok = true;
-        got_any = true;
-      }
-      if (parse_obis_ascii_value(line, "1-0:16.7.0", value)) {
-        data.total_power_w = value;
-        total_p_ok = true;
-        got_any = true;
-      }
-      if (parse_obis_ascii_value(line, "1-0:36.7.0", value)) {
-        data.phase_power_w[0] = value;
-        p1_ok = true;
-        got_any = true;
-      }
-      if (parse_obis_ascii_value(line, "1-0:56.7.0", value)) {
-        data.phase_power_w[1] = value;
-        p2_ok = true;
-        got_any = true;
-      }
-      if (parse_obis_ascii_value(line, "1-0:76.7.0", value)) {
-        data.phase_power_w[2] = value;
-        p3_ok = true;
-        got_any = true;
-      }
-
-      line_len = 0;
-      continue;
-    }
-    if (line_len + 1 < sizeof(line)) {
-      line[line_len++] = c;
-    }
-  }
-
-  data.valid = got_any;
-  if (data.valid) {
+  MeterParseResult parsed = {};
+  bool ok = meter_parser_parse(frame, len, parsed);
+  data.energy_total_kwh = parsed.energy_total_kwh;
+  data.total_power_w = parsed.total_power_w;
+  data.phase_power_w[0] = parsed.phase_power_w[0];
+  data.phase_power_w[1] = parsed.phase_power_w[1];
+  data.phase_power_w[2] = parsed.phase_power_w[2];
+  data.valid = ok;
+  if (ok) {
    g_frames_ok++;
+    g_last_good_frame_ms = millis();
  } else {
    g_frames_parse_fail++;
  }
-  return data.valid;
+
+#if ENABLE_HIL_TRACE
+  if (parsed.meter_seconds_valid) {
+    if (g_meter_seconds_seen && parsed.meter_seconds < g_last_meter_seconds) {
+      hil_trace_eventf("meter_time", "\"classification\":\"rollback\",\"meter_seconds\":%lu,\"previous\":%lu",
+                       static_cast<unsigned long>(parsed.meter_seconds),
+                       static_cast<unsigned long>(g_last_meter_seconds));
+    } else if (g_meter_seconds_seen && parsed.meter_seconds - g_last_meter_seconds > 120) {
+      hil_trace_eventf("meter_time", "\"classification\":\"jump\",\"meter_seconds\":%lu,\"previous\":%lu",
+                       static_cast<unsigned long>(parsed.meter_seconds),
+                       static_cast<unsigned long>(g_last_meter_seconds));
+    }
+    g_last_meter_seconds = parsed.meter_seconds;
+    g_meter_seconds_seen = true;
+  }
+#endif
+
+  hil_trace_eventf("meter_frame", "\"ok\":%s,\"classification\":\"%s\",\"len\":%u,\"fields\":%u,\"meter_seconds\":%lu,\"seconds_valid\":%s",
+                   ok ? "true" : "false", meter_parse_status_text(parsed.status),
+                   static_cast<unsigned>(len), static_cast<unsigned>(parsed.fields_mask),
+                   static_cast<unsigned long>(parsed.meter_seconds),
+                   parsed.meter_seconds_valid ? "true" : "false");
+  return ok;
 }

 bool meter_read(MeterData &data) {
@@ -257,8 +159,6 @@ bool meter_read(MeterData &data) {

  const char *frame = nullptr;
  size_t len = 0;
-  if (!meter_poll_frame(frame, len)) {
-    return false;
-  }
+  if (!meter_poll_frame(frame, len)) return false;
  return meter_parse_frame(frame, len, data);
 }
@@ -0,0 +1,25 @@
+#include "meter_health.h"
+
+MeterHealthEvaluation meter_health_evaluate(MeterHealthState &state,
+                                             bool has_snapshot,
+                                             uint32_t age_ms,
+                                             uint32_t max_age_ms) {
+  MeterHealthEvaluation evaluation = {};
+  evaluation.ok = has_snapshot && age_ms <= max_age_ms;
+  evaluation.classification = evaluation.ok
+      ? MeterHealthClassification::Fresh
+      : (has_snapshot ? MeterHealthClassification::Stale : MeterHealthClassification::NoData);
+  evaluation.fault_started = !evaluation.ok && !state.fault_active;
+  evaluation.recovered = evaluation.ok && state.fault_active;
+  state.fault_active = !evaluation.ok;
+  return evaluation;
+}
+
+const char *meter_health_classification_text(MeterHealthClassification classification) {
+  switch (classification) {
+    case MeterHealthClassification::Fresh: return "fresh";
+    case MeterHealthClassification::NoData: return "no_data";
+    case MeterHealthClassification::Stale: return "stale";
+  }
+  return "unknown";
+}
@@ -0,0 +1,334 @@
+#include "meter_parser.h"
+
+#include <math.h>
+#include <string.h>
+
+namespace {
+
+enum class ObisField : uint8_t {
+  None = 0,
+  Energy,
+  TotalPower,
+  Phase1,
+  Phase2,
+  Phase3,
+  MeterSeconds
+};
+
+struct ParseProgress {
+  bool malformed_value;
+  bool invalid_meter_seconds;
+};
+
+static const char *skip_space(const char *text) {
+  while (*text == ' ' || *text == '\t') {
+    ++text;
+  }
+  return text;
+}
+
+static bool has_obis_prefix(const char *line, const char *obis) {
+  const char *start = skip_space(line);
+  size_t len = strlen(obis);
+  if (strncmp(start, obis, len) != 0) {
+    return false;
+  }
+  char next = start[len];
+  return next == '*' || next == '(';
+}
+
+static ObisField detect_obis_field(const char *line) {
+  if (has_obis_prefix(line, "1-0:1.8.0")) return ObisField::Energy;
+  if (has_obis_prefix(line, "1-0:16.7.0")) return ObisField::TotalPower;
+  if (has_obis_prefix(line, "1-0:36.7.0")) return ObisField::Phase1;
+  if (has_obis_prefix(line, "1-0:56.7.0")) return ObisField::Phase2;
+  if (has_obis_prefix(line, "1-0:76.7.0")) return ObisField::Phase3;
+  if (has_obis_prefix(line, "0-0:96.8.0")) return ObisField::MeterSeconds;
+  return ObisField::None;
+}
+
+static bool parse_decimal(const char *begin, const char *end, float &value) {
+  if (!begin || !end || begin >= end) {
+    return false;
+  }
+  while (begin < end && (*begin == ' ' || *begin == '\t')) ++begin;
+  while (end > begin && (end[-1] == ' ' || end[-1] == '\t')) --end;
+  if (begin >= end) return false;
+
+  bool negative = false;
+  if (*begin == '+' || *begin == '-') {
+    negative = *begin == '-';
+    ++begin;
+  }
+  bool saw_digit = false;
+  bool saw_separator = false;
+  double whole = 0.0;
+  double fraction = 0.0;
+  double divisor = 1.0;
+  for (const char *cur = begin; cur < end; ++cur) {
+    char c = *cur;
+    if (c >= '0' && c <= '9') {
+      saw_digit = true;
+      if (saw_separator) {
+        divisor *= 10.0;
+        fraction += static_cast<double>(c - '0') / divisor;
+      } else {
+        whole = whole * 10.0 + static_cast<double>(c - '0');
+      }
+      continue;
+    }
+    if ((c == '.' || c == ',') && !saw_separator) {
+      saw_separator = true;
+      continue;
+    }
+    return false;
+  }
+  if (!saw_digit) return false;
+  double parsed = whole + fraction;
+  value = static_cast<float>(negative ? -parsed : parsed);
+  return isfinite(value);
+}
+
+static bool payload_bounds(const char *line, const char *&value_begin,
+                           const char *&value_end, const char *&unit_begin,
+                           const char *&unit_end) {
+  const char *lparen = strchr(line, '(');
+  if (!lparen) return false;
+  const char *rparen = strchr(lparen + 1, ')');
+  if (!rparen) return false;
+  const char *asterisk = nullptr;
+  for (const char *cur = lparen + 1; cur < rparen; ++cur) {
+    if (*cur == '*') {
+      asterisk = cur;
+      break;
+    }
+  }
+  value_begin = lparen + 1;
+  value_end = asterisk ? asterisk : rparen;
+  unit_begin = asterisk ? asterisk + 1 : rparen;
+  unit_end = rparen;
+  return true;
+}
+
+static bool unit_equals(const char *begin, const char *end, const char *unit) {
+  while (begin < end && (*begin == ' ' || *begin == '\t')) ++begin;
+  while (end > begin && (end[-1] == ' ' || end[-1] == '\t')) --end;
+  size_t expected = strlen(unit);
+  return static_cast<size_t>(end - begin) == expected && strncmp(begin, unit, expected) == 0;
+}
+
+static bool parse_measurement(const char *line, const char *required_unit,
+                              float &value, bool allow_wh_scale) {
+  const char *value_begin = nullptr;
+  const char *value_end = nullptr;
+  const char *unit_begin = nullptr;
+  const char *unit_end = nullptr;
+  if (!payload_bounds(line, value_begin, value_end, unit_begin, unit_end) ||
+      !parse_decimal(value_begin, value_end, value)) {
+    return false;
+  }
+  if (unit_equals(unit_begin, unit_end, required_unit)) {
+    return true;
+  }
+  if (allow_wh_scale && unit_equals(unit_begin, unit_end, "Wh")) {
+    value *= 0.001f;
+    return true;
+  }
+  return false;
+}
+
+static int8_t hex_nibble(char c) {
+  if (c >= '0' && c <= '9') return static_cast<int8_t>(c - '0');
+  if (c >= 'A' && c <= 'F') return static_cast<int8_t>(10 + c - 'A');
+  if (c >= 'a' && c <= 'f') return static_cast<int8_t>(10 + c - 'a');
+  return -1;
+}
+
+static bool parse_meter_seconds(const char *line, uint32_t &value) {
+  const char *value_begin = nullptr;
+  const char *value_end = nullptr;
+  const char *unit_begin = nullptr;
+  const char *unit_end = nullptr;
+  if (!payload_bounds(line, value_begin, value_end, unit_begin, unit_end)) return false;
+  while (value_begin < value_end && (*value_begin == ' ' || *value_begin == '\t')) ++value_begin;
+  while (value_end > value_begin && (value_end[-1] == ' ' || value_end[-1] == '\t')) --value_end;
+  size_t digits = static_cast<size_t>(value_end - value_begin);
+  if (digits == 0 || digits > 8) return false;
+  uint32_t parsed = 0;
+  for (const char *cur = value_begin; cur < value_end; ++cur) {
+    int8_t nibble = hex_nibble(*cur);
+    if (nibble < 0) return false;
+    parsed = (parsed << 4) | static_cast<uint32_t>(nibble);
+  }
+  value = parsed;
+  return true;
+}
+
+static void parse_line(const char *line, MeterParseResult &result, ParseProgress &progress) {
+  ObisField field = detect_obis_field(line);
+  float value = NAN;
+  bool ok = false;
+  switch (field) {
+    case ObisField::Energy:
+      ok = parse_measurement(line, "kWh", value, true);
+      if (ok) {
+        result.energy_total_kwh = value;
+        result.fields_mask |= 0x01;
+      }
+      break;
+    case ObisField::TotalPower:
+      ok = parse_measurement(line, "W", value, false);
+      if (ok) {
+        result.total_power_w = value;
+        result.fields_mask |= 0x02;
+      }
+      break;
+    case ObisField::Phase1:
+    case ObisField::Phase2:
+    case ObisField::Phase3: {
+      ok = parse_measurement(line, "W", value, false);
+      if (ok) {
+        uint8_t index = field == ObisField::Phase1 ? 0 : (field == ObisField::Phase2 ? 1 : 2);
+        result.phase_power_w[index] = value;
+        result.fields_mask |= static_cast<uint8_t>(0x04 << index);
+      }
+      break;
+    }
+    case ObisField::MeterSeconds:
+      ok = parse_meter_seconds(line, result.meter_seconds);
+      result.meter_seconds_valid = ok;
+      if (!ok) progress.invalid_meter_seconds = true;
+      return;
+    case ObisField::None:
+      return;
+  }
+  if (!ok) progress.malformed_value = true;
+}
+
+}  // namespace
+
+void meter_frame_collector_init(MeterFrameCollector &collector) {
+  collector.buffer[0] = '\0';
+  collector.length = 0;
+  collector.last_rx_ms = 0;
+  collector.in_frame = false;
+}
+
+MeterFrameEvent meter_frame_collector_check_timeout(MeterFrameCollector &collector,
+                                                     uint32_t now_ms,
+                                                     uint32_t timeout_ms) {
+  if (!collector.in_frame || now_ms - collector.last_rx_ms <= timeout_ms) {
+    return MeterFrameEvent::None;
+  }
+  collector.length = 0;
+  collector.buffer[0] = '\0';
+  collector.in_frame = false;
+  return MeterFrameEvent::Timeout;
+}
+
+MeterFrameEvent meter_frame_collector_feed(MeterFrameCollector &collector,
+                                            char byte,
+                                            uint32_t now_ms) {
+  if (!collector.in_frame) {
+    if (byte != '/') return MeterFrameEvent::None;
+    collector.in_frame = true;
+    collector.length = 1;
+    collector.buffer[0] = '/';
+    collector.buffer[1] = '\0';
+    collector.last_rx_ms = now_ms;
+    return MeterFrameEvent::None;
+  }
+
+  collector.last_rx_ms = now_ms;
+  if (byte == '/') {
+    collector.length = 1;
+    collector.buffer[0] = '/';
+    collector.buffer[1] = '\0';
+    return MeterFrameEvent::Resync;
+  }
+  if (collector.length >= METER_FRAME_MAX) {
+    collector.length = 0;
+    collector.buffer[0] = '\0';
+    collector.in_frame = false;
+    return MeterFrameEvent::Overflow;
+  }
+  collector.buffer[collector.length++] = byte;
+  collector.buffer[collector.length] = '\0';
+  if (byte == '!') {
+    collector.in_frame = false;
+    return MeterFrameEvent::Complete;
+  }
+  return MeterFrameEvent::None;
+}
+
+bool meter_parser_parse(const char *frame, size_t len, MeterParseResult &result) {
+  result = {};
+  result.energy_total_kwh = NAN;
+  result.total_power_w = NAN;
+  result.phase_power_w[0] = NAN;
+  result.phase_power_w[1] = NAN;
+  result.phase_power_w[2] = NAN;
+  result.status = MeterParseStatus::Empty;
+  if (!frame || len == 0 || frame[0] != '/') return false;
+
+  char line[128];
+  size_t line_len = 0;
+  bool terminator = false;
+  ParseProgress progress = {};
+  for (size_t i = 0; i < len; ++i) {
+    char c = frame[i];
+    if (c == '\r') continue;
+    if (c == '!') {
+      if (line_len > 0) {
+        line[line_len] = '\0';
+        parse_line(line, result, progress);
+      }
+      terminator = true;
+      break;
+    }
+    if (c == '\n') {
+      line[line_len] = '\0';
+      parse_line(line, result, progress);
+      line_len = 0;
+      continue;
+    }
+    if (line_len + 1 >= sizeof(line)) {
+      result.status = MeterParseStatus::LineTooLong;
+      return false;
+    }
+    line[line_len++] = c;
+  }
+
+  if (!terminator) {
+    result.status = MeterParseStatus::MissingTerminator;
+    return false;
+  }
+  if (progress.invalid_meter_seconds) {
+    result.status = MeterParseStatus::InvalidMeterSeconds;
+    return false;
+  }
+  if (progress.malformed_value) {
+    result.status = MeterParseStatus::MalformedValue;
+    return false;
+  }
+  if (result.fields_mask != METER_REQUIRED_FIELDS_MASK || !result.meter_seconds_valid) {
+    result.status = MeterParseStatus::MissingRequired;
+    return false;
+  }
+  result.valid = true;
+  result.status = MeterParseStatus::Valid;
+  return true;
+}
+
+const char *meter_parse_status_text(MeterParseStatus status) {
+  switch (status) {
+    case MeterParseStatus::Valid: return "valid";
+    case MeterParseStatus::Empty: return "empty";
+    case MeterParseStatus::MissingTerminator: return "missing_terminator";
+    case MeterParseStatus::LineTooLong: return "line_too_long";
+    case MeterParseStatus::MissingRequired: return "missing_required";
+    case MeterParseStatus::InvalidMeterSeconds: return "invalid_meter_seconds";
+    case MeterParseStatus::MalformedValue: return "malformed_value";
+  }
+  return "unknown";
+}
@@ -0,0 +1,21 @@
+#!/usr/bin/env python3
+import argparse
+
+from hil_common import load_config_if_present, main_guard, new_artifact_dir, pio_command, run_checked
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description="Build the configured HIL firmware without accessing serial ports")
+    parser.add_argument("--artifact-dir")
+    args = parser.parse_args()
+    config = load_config_if_present()
+    environment = config.get("hil", {}).get("environment", "hil")
+    artifact = new_artifact_dir("build") if not args.artifact_dir else __import__("pathlib").Path(args.artifact_dir)
+    artifact.mkdir(parents=True, exist_ok=True)
+    run_checked([pio_command(), "run", "-e", environment], log=artifact / "build.log")
+    print(f"Build log: {artifact / 'build.log'}")
+    return 0
+
+
+if __name__ == "__main__":
+    main_guard(main)
@@ -0,0 +1,38 @@
+#!/usr/bin/env python3
+import argparse
+from pathlib import Path
+
+from hil_common import PairCapture, load_config, main_guard, new_artifact_dir, pair_ports
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description="Capture both approved serial streams concurrently")
+    parser.add_argument("--seconds", type=float)
+    parser.add_argument("--artifact-dir")
+    parser.add_argument("--reset", action="store_true", help="normally reboot both boards after capture starts")
+    parser.add_argument("--fault", choices=["drop_chunk", "duplicate_chunk", "corrupt_chunk", "suppress_ack",
+                                                   "delay_ack", "wrong_ack_batch"])
+    parser.add_argument("--fault-after", type=float, default=10.0)
+    args = parser.parse_args()
+    config = load_config()
+    ports = pair_ports(config, "access")
+    baud = int(config.get("hil", {}).get("baud", 115200))
+    seconds = args.seconds or float(config.get("hil", {}).get("capture_seconds", 180))
+    artifact = Path(args.artifact_dir) if args.artifact_dir else new_artifact_dir("capture")
+    sent = False
+
+    def tick(elapsed: float, capture: PairCapture) -> None:
+        nonlocal sent
+        if args.fault and not sent and elapsed >= args.fault_after:
+            target = "sender" if args.fault.endswith("chunk") else "receiver"
+            command = "delay_ack:1000" if args.fault == "delay_ack" else args.fault
+            capture.send_fault(target, command)
+            sent = True
+
+    PairCapture(ports, baud, artifact).run(seconds, tick, reset_on_start=args.reset)
+    print(f"Capture artifacts: {artifact}")
+    return 0
+
+
+if __name__ == "__main__":
+    main_guard(main)
@@ -0,0 +1,21 @@
+#!/usr/bin/env python3
+from pathlib import Path
+
+from hil_common import main_guard, pio_command, run_checked
+
+
+def main() -> int:
+    print("Stable serial candidates (listing only; no device is opened):")
+    by_id = Path("/dev/serial/by-id")
+    candidates = sorted(by_id.iterdir()) if by_id.is_dir() else []
+    if not candidates:
+        print("  none")
+    for candidate in candidates:
+        print(f"  {candidate} -> {candidate.resolve()}")
+    print("\nPlatformIO discovery (metadata only):")
+    run_checked([pio_command(), "device", "list"])
+    return 0
+
+
+if __name__ == "__main__":
+    main_guard(main)
@@ -0,0 +1,33 @@
+#!/usr/bin/env python3
+import argparse
+from pathlib import Path
+
+from hil_common import HilError, load_config, main_guard, new_artifact_dir, pair_ports, pio_command, run_checked
+
+
+def flash_pair(artifact: Path, yes_flash: bool) -> None:
+    if not yes_flash:
+        raise HilError("refusing to flash without --yes-flash")
+    config = load_config()
+    ports = pair_ports(config, "flash")
+    environment = config.get("hil", {}).get("environment", "hil")
+    artifact.mkdir(parents=True, exist_ok=True)
+    run_checked([pio_command(), "run", "-e", environment], log=artifact / "build.log")
+    for port in ports:
+        run_checked([pio_command(), "run", "-e", environment, "-t", "upload", "--upload-port", str(port.device)],
+                    log=artifact / f"flash-{port.name}.log")
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description="Build once and flash the explicitly approved board pair")
+    parser.add_argument("--yes-flash", action="store_true", help="confirm the approved_flash entries for this invocation")
+    parser.add_argument("--artifact-dir")
+    args = parser.parse_args()
+    artifact = Path(args.artifact_dir) if args.artifact_dir else new_artifact_dir("flash")
+    flash_pair(artifact, args.yes_flash)
+    print(f"Flash logs: {artifact}")
+    return 0
+
+
+if __name__ == "__main__":
+    main_guard(main)
@@ -0,0 +1,261 @@
+from __future__ import annotations
+
+import json
+import os
+import shutil
+import subprocess
+import sys
+import threading
+import time
+from dataclasses import dataclass
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any, Callable
+
+try:
+    import tomllib
+except ModuleNotFoundError:  # pragma: no cover - Python >=3.11 is documented
+    tomllib = None
+
+
+ROOT = Path(__file__).resolve().parents[2]
+DEFAULT_CONFIG = ROOT / ".hil" / "local.toml"
+ARTIFACT_ROOT = ROOT / "artifacts" / "hil"
+HIL_PREFIX = b"HIL:"
+
+
+class HilError(RuntimeError):
+    pass
+
+
+@dataclass(frozen=True)
+class PortConfig:
+    name: str
+    device: Path
+    approved_access: bool
+    approved_flash: bool = False
+
+
+def utc_stamp() -> str:
+    return datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
+
+
+def new_artifact_dir(label: str = "run") -> Path:
+    path = ARTIFACT_ROOT / f"{utc_stamp()}-{label}"
+    suffix = 1
+    while path.exists():
+        path = ARTIFACT_ROOT / f"{utc_stamp()}-{label}-{suffix}"
+        suffix += 1
+    path.mkdir(parents=True)
+    return path
+
+
+def load_config(path: Path = DEFAULT_CONFIG) -> dict[str, Any]:
+    if tomllib is None:
+        raise HilError("Python 3.11 or newer is required (tomllib is unavailable)")
+    if not path.is_file():
+        raise HilError(f"missing {path}; copy .hil/local.example.toml and enter only approved ports")
+    with path.open("rb") as handle:
+        config = tomllib.load(handle)
+    return config
+
+
+def load_config_if_present(path: Path = DEFAULT_CONFIG) -> dict[str, Any]:
+    return load_config(path) if path.is_file() else {}
+
+
+def configured_port(config: dict[str, Any], name: str, operation: str) -> PortConfig:
+    raw = config.get("ports", {}).get(name, {})
+    device_text = str(raw.get("device", "")).strip()
+    if not device_text:
+        raise HilError(f"ports.{name}.device is not configured")
+    device = Path(device_text)
+    if not str(device).startswith("/dev/serial/by-id/"):
+        raise HilError(f"{name} must use a stable /dev/serial/by-id path, got {device}")
+    if not device.exists():
+        raise HilError(f"configured {name} device does not exist: {device}")
+    access = raw.get("approved_access") is True
+    flash = raw.get("approved_flash") is True
+    if operation in {"access", "flash"} and not access:
+        raise HilError(f"first access to {device} is not approved in {DEFAULT_CONFIG}")
+    if operation == "flash" and not flash:
+        raise HilError(f"flashing {device} is not approved in {DEFAULT_CONFIG}")
+    return PortConfig(name, device, access, flash)
+
+
+def pair_ports(config: dict[str, Any], operation: str) -> tuple[PortConfig, PortConfig]:
+    sender = configured_port(config, "sender", operation)
+    receiver = configured_port(config, "receiver", operation)
+    if sender.device.resolve() == receiver.device.resolve():
+        raise HilError("sender and receiver resolve to the same device")
+    return sender, receiver
+
+
+def pio_command() -> str:
+    for candidate in (shutil.which("pio"), shutil.which("platformio")):
+        if candidate:
+            return candidate
+    local = Path.home() / ".platformio" / "penv" / "bin" / "pio"
+    if local.is_file() and os.access(local, os.X_OK):
+        return str(local)
+    raise HilError("PlatformIO CLI was not found on PATH or in ~/.platformio/penv/bin/pio")
+
+
+def run_checked(command: list[str], *, cwd: Path = ROOT, log: Path | None = None) -> str:
+    process = subprocess.run(command, cwd=cwd, text=True, stdout=subprocess.PIPE,
+                             stderr=subprocess.STDOUT, check=False)
+    if log:
+        log.parent.mkdir(parents=True, exist_ok=True)
+        log.write_text(process.stdout, encoding="utf-8")
+    sys.stdout.write(process.stdout)
+    if process.returncode:
+        raise HilError(f"command failed ({process.returncode}): {' '.join(command)}")
+    return process.stdout
+
+
+def parse_hil_line(line: bytes) -> dict[str, Any] | None:
+    marker = line.find(HIL_PREFIX)
+    if marker < 0:
+        return None
+    try:
+        value = json.loads(line[marker + len(HIL_PREFIX):].decode("utf-8", "strict").strip())
+    except (UnicodeDecodeError, json.JSONDecodeError):
+        return None
+    return value if isinstance(value, dict) and isinstance(value.get("event"), str) else None
+
+
+def serial_module():
+    try:
+        import serial
+    except ModuleNotFoundError as exc:
+        raise HilError("pyserial is required; install requirements-hil.txt in a project-local virtualenv") from exc
+    return serial
+
+
+class PairCapture:
+    def __init__(self, ports: tuple[PortConfig, PortConfig], baud: int, output: Path):
+        self.ports = ports
+        self.baud = baud
+        self.output = output
+        self.stop = threading.Event()
+        self.events: list[dict[str, Any]] = []
+        self._lock = threading.Lock()
+        self._handles: dict[str, Any] = {}
+        self._threads: list[threading.Thread] = []
+
+    def _reader(self, port: PortConfig) -> None:
+        raw_path = self.output / f"{port.name}.raw"
+        log_path = self.output / f"{port.name}.log"
+        handle = self._handles[port.name]
+        with raw_path.open("ab", buffering=0) as raw, log_path.open("a", encoding="utf-8", buffering=1) as log:
+            while not self.stop.is_set():
+                line = handle.readline()
+                if not line:
+                    continue
+                host_ns = time.time_ns()
+                raw.write(line)
+                rendered = line.decode("utf-8", "replace").rstrip("\r\n")
+                log.write(f"{datetime.fromtimestamp(host_ns / 1e9, timezone.utc).isoformat()} {rendered}\n")
+                event = parse_hil_line(line)
+                if event is not None:
+                    event["host_time_ns"] = host_ns
+                    event["source"] = port.name
+                    with self._lock:
+                        self.events.append(event)
+
+    def start(self) -> None:
+        serial = serial_module()
+        self.output.mkdir(parents=True, exist_ok=True)
+        try:
+            for port in self.ports:
+                # Opening is deliberately delayed until approval checks have completed.
+                self._handles[port.name] = serial.Serial(str(port.device), self.baud, timeout=0.25,
+                                                         write_timeout=1, exclusive=True)
+            for port in self.ports:
+                thread = threading.Thread(target=self._reader, args=(port,), daemon=True,
+                                          name=f"hil-{port.name}")
+                thread.start()
+                self._threads.append(thread)
+        except Exception:
+            self.close()
+            raise
+
+    def send_fault(self, target: str, fault: str) -> None:
+        allowed = {"drop_chunk", "duplicate_chunk", "corrupt_chunk", "suppress_ack",
+                   "wrong_ack_batch", "clear"}
+        if fault not in allowed and not fault.startswith("delay_ack:"):
+            raise HilError(f"unsupported HIL fault: {fault}")
+        handle = self._handles.get(target)
+        if handle is None:
+            raise HilError(f"serial target is not open: {target}")
+        handle.write(f"HILCMD:{fault}\n".encode("ascii"))
+        handle.flush()
+
+    def reset_pair(self) -> None:
+        """Pulse ESP32 EN through the normal USB auto-reset circuit.
+
+        DTR remains deasserted so GPIO0 is not held low and the boards boot the
+        flashed application rather than the ROM download mode.
+        """
+        for handle in self._handles.values():
+            handle.dtr = False
+            handle.rts = True
+        time.sleep(0.15)
+        for handle in self._handles.values():
+            handle.rts = False
+        time.sleep(0.25)
+
+    def close(self) -> None:
+        self.stop.set()
+        for thread in self._threads:
+            thread.join(timeout=2)
+        for handle in self._handles.values():
+            try:
+                handle.close()
+            except Exception:
+                pass
+        self._handles.clear()
+        ordered = sorted(self.events, key=lambda item: item["host_time_ns"])
+        with (self.output / "events.jsonl").open("w", encoding="utf-8") as stream:
+            for event in ordered:
+                stream.write(json.dumps(event, separators=(",", ":"), sort_keys=True) + "\n")
+
+    def run(self, seconds: float, on_tick: Callable[[float, "PairCapture"], None] | None = None,
+            reset_on_start: bool = False) -> None:
+        self.start()
+        if reset_on_start:
+            self.reset_pair()
+        started = time.monotonic()
+        try:
+            while time.monotonic() - started < seconds and not self.stop.is_set():
+                elapsed = time.monotonic() - started
+                if on_tick:
+                    on_tick(elapsed, self)
+                time.sleep(0.2)
+        except KeyboardInterrupt:
+            pass
+        finally:
+            self.close()
+
+
+def read_events(artifact: Path) -> list[dict[str, Any]]:
+    path = artifact / "events.jsonl"
+    if not path.is_file():
+        return []
+    events = []
+    for line in path.read_text(encoding="utf-8").splitlines():
+        try:
+            item = json.loads(line)
+        except json.JSONDecodeError:
+            continue
+        if isinstance(item, dict):
+            events.append(item)
+    return events
+
+
+def main_guard(function: Callable[[], int]) -> None:
+    try:
+        raise SystemExit(function())
+    except HilError as exc:
+        print(f"HIL error: {exc}", file=sys.stderr)
+        raise SystemExit(2)
@@ -0,0 +1,68 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import argparse
+import time
+
+from hil_common import configured_port, load_config, main_guard, serial_module
+
+
+def frame(seconds: str = "00000064", include_energy: bool = True, terminator: bool = True) -> bytes:
+    lines = ["/DD3HIL", f"0-0:96.8.0*255({seconds})"]
+    if include_energy:
+        lines.append("1-0:1.8.0*255(001234.567*kWh)")
+    lines.extend([
+        "1-0:16.7.0*255(000950*W)",
+        "1-0:36.7.0*255(000500*W)",
+        "1-0:56.7.0*255(000450*W)",
+        "1-0:76.7.0*255(000000*W)",
+    ])
+    text = "\r\n".join(lines) + "\r\n"
+    if terminator:
+        text += "!\r\n"
+    return text.encode("ascii")
+
+
+def fixture(name: str) -> list[tuple[bytes, float]]:
+    valid = frame()
+    fixtures = {
+        "valid": [(valid, 0)],
+        "timeout": [(b"/DD3HIL\r\n1-0:1.8.0*255(1*kWh)", 2.0)],
+        "truncated": [(valid[:40], 2.0)],
+        "missing_terminator": [(frame(terminator=False), 2.0)],
+        "malformed_obis": [(valid.replace(b"1-0:1.8.0", b"1-0:X.Y.Z"), 0)],
+        "missing_required": [(frame(include_energy=False), 0)],
+        "invalid_meter_seconds": [(frame(seconds="GGGGGGGG"), 0)],
+        "timestamp_rollback": [(frame(seconds="00000100"), 0.2), (frame(seconds="00000080"), 0)],
+        "timestamp_jump": [(frame(seconds="00000100"), 0.2), (frame(seconds="00001000"), 0)],
+        "oversized": [(b"/" + b"A" * 600 + b"!\r\n", 0)],
+        "malformed_then_valid": [(valid.replace(b"1-0:1.8.0", b"1-0:X.Y.Z"), 0.2), (valid, 0)],
+    }
+    if name == "slow_inter_byte":
+        return [(bytes([byte]), 0.02) for byte in valid]
+    return fixtures[name]
+
+
+def main() -> int:
+    choices = ["valid", "timeout", "truncated", "missing_terminator", "malformed_obis",
+               "missing_required", "invalid_meter_seconds", "timestamp_rollback", "timestamp_jump",
+               "oversized", "slow_inter_byte", "malformed_then_valid"]
+    parser = argparse.ArgumentParser(description="Drive an explicitly approved isolated 9600 7E1 meter simulator")
+    parser.add_argument("fixture", choices=choices)
+    args = parser.parse_args()
+    config = load_config()
+    port = configured_port(config, "meter_simulator", "access")
+    serial = serial_module()
+    with serial.Serial(str(port.device), 9600, bytesize=serial.SEVENBITS, parity=serial.PARITY_EVEN,
+                       stopbits=serial.STOPBITS_ONE, timeout=1, write_timeout=2, exclusive=True) as handle:
+        for payload, pause in fixture(args.fixture):
+            handle.write(payload)
+            handle.flush()
+            if pause:
+                time.sleep(pause)
+    print(f"Sent {args.fixture} through approved isolated simulator {port.device}")
+    return 0
+
+
+if __name__ == "__main__":
+    main_guard(main)
@@ -0,0 +1,157 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import argparse
+import json
+from dataclasses import dataclass
+from pathlib import Path
+from xml.etree import ElementTree as ET
+
+from hil_common import HilError, main_guard, read_events
+
+
+@dataclass
+class Check:
+    name: str
+    status: str
+    detail: str
+
+
+def _events(events, name, source=None):
+    return [event for event in events if event.get("event") == name and (source is None or event.get("source") == source)]
+
+
+def evaluate(events: list[dict], requested_fault: str | list[str] | None = None) -> list[Check]:
+    checks: list[Check] = []
+    roles = {event.get("source"): event.get("role") for event in _events(events, "role")}
+    role_ok = roles.get("sender") == "sender" and roles.get("receiver") == "receiver"
+    checks.append(Check("role mapping", "pass" if role_ok else "fail", json.dumps(roles, sort_keys=True)))
+
+    boot_counts = {source: len(_events(events, "boot", source)) for source in ("sender", "receiver")}
+    reset_ok = all(1 <= count <= 2 for count in boot_counts.values())
+    checks.append(Check("no reset loop", "pass" if reset_ok else "fail", json.dumps(boot_counts, sort_keys=True)))
+
+    time_ok = any(event.get("stage") == "complete" and event.get("ok") is True
+                  for event in _events(events, "time_bootstrap", "sender"))
+    checks.append(Check("time bootstrap", "pass" if time_ok else "fail", "sender completion event captured" if time_ok else "no sender completion event"))
+
+    meter = _events(events, "meter_frame", "sender")
+    meter_ok = any(event.get("ok") is True for event in meter)
+    meter_status = "pass" if meter_ok else "blocked"
+    checks.append(Check("meter parser", meter_status,
+                        "valid or partial meter frame captured" if meter_ok else "no valid meter evidence; meter/safe simulator unavailable or silent"))
+
+    health = _events(events, "health")
+    checks.append(Check("runtime diagnostics", "pass" if len(health) >= 2 else "fail", f"{len(health)} health events"))
+
+    created = {event.get("batch_id") for event in _events(events, "batch_created", "sender")}
+    complete = {event.get("batch_id") for event in _events(events, "reassembly_complete", "receiver")}
+    decoded = {event.get("batch_id") for event in _events(events, "payload_decode", "receiver") if event.get("ok") is True}
+    ack_tx = {event.get("batch_id") for event in _events(events, "ack_tx", "receiver") if event.get("ok") is True}
+    ack_rx = {event.get("batch_id") for event in _events(events, "ack_rx", "sender") if event.get("ok") is True}
+    correlated = sorted((created & complete & decoded & ack_tx & ack_rx) - {None})
+    if correlated:
+        batch_status = "pass"
+        batch_detail = f"correlated batch IDs: {correlated}"
+    elif not created and not meter_ok:
+        batch_status = "blocked"
+        batch_detail = "no valid meter input, so publication was correctly suppressed"
+    else:
+        batch_status = "fail"
+        batch_detail = f"correlated batch IDs: {correlated}"
+    checks.append(Check("batch and ACK correlation", batch_status, batch_detail))
+
+    requested_faults = ([requested_fault] if isinstance(requested_fault, str) else requested_fault) or []
+    for fault in requested_faults:
+        injected = [event for event in _events(events, "fault_injected") if event.get("fault") == fault]
+        injection_ns = min((event.get("host_time_ns", 0) for event in injected), default=0)
+        later_injections = [event.get("host_time_ns", 0) for event in _events(events, "fault_injected")
+                            if event.get("host_time_ns", 0) > injection_ns]
+        window_end = min(later_injections) if later_injections else 2**63 - 1
+        in_window = lambda event: injection_ns < event.get("host_time_ns", 0) < window_end
+        recovered = any(event.get("event") == "ack_rx" and event.get("ok") is True and in_window(event)
+                        for event in events)
+        details = [f"injected={bool(injected)}", f"later successful ACK={recovered}"]
+        expected = True
+        if fault in {"drop_chunk", "corrupt_chunk", "suppress_ack", "wrong_ack_batch"}:
+            retry = any(event.get("event") == "retry" and in_window(event) for event in events)
+            details.append(f"retry={retry}")
+            expected = retry
+            if fault in {"drop_chunk", "corrupt_chunk"} and retry:
+                first_retry_ns = min(event.get("host_time_ns", 0) for event in events
+                                     if event.get("event") == "retry" and in_window(event))
+                premature_decode = any(event.get("event") == "payload_decode" and event.get("ok") is True and
+                                       injection_ns < event.get("host_time_ns", 0) < first_retry_ns
+                                       for event in events)
+                details.append(f"decoded before retry={premature_decode}")
+                expected = expected and not premature_decode
+        if fault == "duplicate_chunk":
+            duplicate_detected = any(event.get("event") == "duplicate" and in_window(event) for event in events)
+            details.append(f"receiver duplicate detection={duplicate_detected}")
+            expected = duplicate_detected
+        if fault == "corrupt_chunk":
+            crc_reject = any(event.get("event") == "lora_reject" and event.get("reason") == "crc_fail" and
+                             in_window(event) for event in events)
+            details.append(f"CRC rejection={crc_reject}")
+            expected = expected and crc_reject
+        if fault == "wrong_ack_batch":
+            wrong_reject = any(event.get("event") == "ack_rx" and event.get("ok") is False and
+                               in_window(event) for event in events)
+            details.append(f"wrong ACK rejected={wrong_reject}")
+            expected = expected and wrong_reject
+        passed = bool(injected) and recovered and expected
+        checks.append(Check(f"fault {fault}", "pass" if passed else "fail", ", ".join(details)))
+    return checks
+
+
+def write_report(artifact: Path, checks: list[Check], events: list[dict]) -> None:
+    lines = ["# DD3 local HIL report", "", f"Artifact directory: `{artifact}`", "", "## Results", ""]
+    for check in checks:
+        lines.append(f"- **{check.status.upper()}** — {check.name}: {check.detail}")
+    lines.extend(["", "## Evidence summary", ""])
+    resets = [{"source": e.get("source"), "reset_reason": e.get("reset_reason")} for e in _events(events, "boot")]
+    lines.append(f"- Reset evidence: `{json.dumps(resets, sort_keys=True)}`")
+    lines.append(f"- Meter classifications: `{json.dumps([e.get('classification') for e in _events(events, 'meter_frame')])}`")
+    lines.append(f"- Retry events: `{len(_events(events, 'retry'))}`")
+    queue_drops = [event for event in _events(events, "queue_drop") if event.get("inflight") is not True]
+    lines.append(f"- Queue drops: `{len(queue_drops)}`")
+    heaps = [e.get("heap_free") for e in _events(events, "health") if isinstance(e.get("heap_free"), int)]
+    lines.append(f"- Minimum observed free heap: `{min(heaps) if heaps else 'not captured'}`")
+    lines.extend(["", "Raw serial bytes and timestamped text logs are retained beside this report.", ""])
+    (artifact / "report.md").write_text("\n".join(lines), encoding="utf-8")
+
+    suite = ET.Element("testsuite", name="dd3-hil", tests=str(len(checks)),
+                       failures=str(sum(c.status == "fail" for c in checks)),
+                       skipped=str(sum(c.status == "blocked" for c in checks)))
+    for check in checks:
+        case = ET.SubElement(suite, "testcase", name=check.name, classname="hil")
+        if check.status == "fail":
+            ET.SubElement(case, "failure", message=check.detail)
+        elif check.status == "blocked":
+            ET.SubElement(case, "skipped", message=check.detail)
+        ET.SubElement(case, "system-out").text = check.detail
+    ET.ElementTree(suite).write(artifact / "junit.xml", encoding="utf-8", xml_declaration=True)
+
+
+def generate(artifact: Path, requested_fault: str | list[str] | None = None) -> list[Check]:
+    events = read_events(artifact)
+    if not events:
+        raise HilError(f"no parsed HIL events in {artifact / 'events.jsonl'}")
+    checks = evaluate(events, requested_fault)
+    write_report(artifact, checks, events)
+    return checks
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description="Generate Markdown and JUnit reports from captured HIL evidence")
+    parser.add_argument("artifact_dir", type=Path)
+    parser.add_argument("--fault", action="append")
+    args = parser.parse_args()
+    checks = generate(args.artifact_dir, args.fault)
+    for check in checks:
+        print(f"{check.status.upper():7} {check.name}: {check.detail}")
+    return 1 if any(check.status == "fail" for check in checks) else 0
+
+
+if __name__ == "__main__":
+    main_guard(main)
@@ -0,0 +1,39 @@
+#!/usr/bin/env python3
+import argparse
+from flash_pair import flash_pair
+from hil_common import PairCapture, load_config, main_guard, new_artifact_dir, pair_ports
+from report import generate
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description="Build, flash, capture, and verify one DD3 HIL run")
+    parser.add_argument("--yes-flash", action="store_true")
+    parser.add_argument("--seconds", type=float)
+    parser.add_argument("--fault", choices=["drop_chunk", "duplicate_chunk", "corrupt_chunk", "suppress_ack",
+                                                   "delay_ack", "wrong_ack_batch"])
+    parser.add_argument("--fault-after", type=float, default=40.0)
+    args = parser.parse_args()
+    config = load_config()
+    ports = pair_ports(config, "flash")
+    artifact = new_artifact_dir("baseline" if not args.fault else f"fault-{args.fault}")
+    flash_pair(artifact, args.yes_flash)
+    baud = int(config.get("hil", {}).get("baud", 115200))
+    seconds = args.seconds or float(config.get("hil", {}).get("capture_seconds", 180))
+    sent = False
+
+    def tick(elapsed, capture):
+        nonlocal sent
+        if args.fault and not sent and elapsed >= args.fault_after:
+            target = "sender" if args.fault.endswith("chunk") else "receiver"
+            command = "delay_ack:1000" if args.fault == "delay_ack" else args.fault
+            capture.send_fault(target, command)
+            sent = True
+
+    PairCapture(ports, baud, artifact).run(seconds, tick, reset_on_start=True)
+    checks = generate(artifact, args.fault)
+    print(f"HIL report: {artifact / 'report.md'}")
+    return 1 if any(check.status == "fail" for check in checks) else 0
+
+
+if __name__ == "__main__":
+    main_guard(main)
@@ -0,0 +1,104 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import argparse
+import json
+import time
+
+from hil_common import PairCapture, load_config, main_guard, new_artifact_dir, pair_ports
+from report import generate
+
+
+FAULTS = [
+    "drop_chunk",
+    "duplicate_chunk",
+    "corrupt_chunk",
+    "suppress_ack",
+    "delay_ack",
+    "wrong_ack_batch",
+]
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description="Run all one-shot LoRa HIL faults in one approved capture")
+    parser.add_argument("--fault", action="append", choices=FAULTS,
+                        help="run only this fault (repeatable); defaults to all")
+    parser.add_argument("--seconds", type=float, default=600)
+    parser.add_argument("--arm-after", type=float, default=10)
+    parser.add_argument("--case-timeout", type=float, default=100)
+    args = parser.parse_args()
+    faults = args.fault or FAULTS
+    config = load_config()
+    ports = pair_ports(config, "access")
+    baud = int(config.get("hil", {}).get("baud", 115200))
+    artifact = new_artifact_dir("fault-suite")
+    state = {"index": 0, "armed_at": None, "last_command_at": None, "armed_confirmed": False,
+             "injected_ns": None, "recovered_at": None, "cases": []}
+
+    def arm(capture: PairCapture, now: float) -> None:
+        fault = faults[state["index"]]
+        target = "sender" if fault.endswith("chunk") else "receiver"
+        command = "delay_ack:1000" if fault == "delay_ack" else fault
+        capture.send_fault(target, command)
+        state["armed_at"] = now
+        state["last_command_at"] = now
+        state["armed_confirmed"] = False
+        state["injected_ns"] = None
+        state["recovered_at"] = None
+        print(f"Armed {fault} on {target}", flush=True)
+
+    def finish_case(capture: PairCapture, now: float, recovered: bool) -> None:
+        fault = faults[state["index"]]
+        state["cases"].append({"fault": fault, "recovered": recovered})
+        state["index"] += 1
+        if state["index"] >= len(faults):
+            capture.stop.set()
+            return
+        arm(capture, now)
+
+    def tick(elapsed: float, capture: PairCapture) -> None:
+        now = time.monotonic()
+        if state["index"] >= len(faults):
+            capture.stop.set()
+            return
+        if state["armed_at"] is None:
+            if elapsed >= args.arm_after:
+                arm(capture, now)
+            return
+        fault = faults[state["index"]]
+        snapshot = list(capture.events)
+        armed = [event for event in snapshot if event.get("event") == "fault_armed" and
+                 event.get("fault") == fault]
+        if armed:
+            state["armed_confirmed"] = True
+        if not state["armed_confirmed"] and now - state["last_command_at"] >= 5:
+            target = "sender" if fault.endswith("chunk") else "receiver"
+            command = "delay_ack:1000" if fault == "delay_ack" else fault
+            capture.send_fault(target, command)
+            state["last_command_at"] = now
+            print(f"Re-sent {fault}; awaiting device arm acknowledgement", flush=True)
+        injections = [event for event in snapshot if event.get("event") == "fault_injected" and
+                      event.get("fault") == fault]
+        if injections and state["injected_ns"] is None:
+            state["injected_ns"] = min(event.get("host_time_ns", 0) for event in injections)
+        if state["injected_ns"]:
+            recovered = any(event.get("event") == "ack_rx" and event.get("ok") is True and
+                            event.get("host_time_ns", 0) > state["injected_ns"] for event in snapshot)
+            if recovered:
+                if state["recovered_at"] is None:
+                    state["recovered_at"] = now
+                elif now - state["recovered_at"] >= 3:
+                    finish_case(capture, now, True)
+                    return
+        if now - state["armed_at"] >= args.case_timeout:
+            finish_case(capture, now, False)
+
+    PairCapture(ports, baud, artifact).run(args.seconds, tick, reset_on_start=True)
+    (artifact / "fault-suite.json").write_text(json.dumps(state["cases"], indent=2) + "\n", encoding="utf-8")
+    checks = generate(artifact, faults)
+    print(f"Fault-suite report: {artifact / 'report.md'}")
+    return 1 if any(check.status == "fail" for check in checks) else 0
+
+
+if __name__ == "__main__":
+    main_guard(main)