From 380084029a3e4704d948958b343a5a0af1eef645 Mon Sep 17 00:00:00 2001 From: acidburns Date: Sat, 21 Feb 2026 01:01:39 +0100 Subject: [PATCH] Add comprehensive README for workspace functionality and implementation --- README.md | 237 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 237 insertions(+) create mode 100644 README.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..0e529e3 --- /dev/null +++ b/README.md @@ -0,0 +1,237 @@ +# DD3 LoRa Bridge Rust Port (Host-First, Test-First) + +This repository is a Rust workspace port of the DD3 LoRa Bridge with compatibility as the primary goal. + +The implementation is organized so protocol, contracts, and state-machine behavior can be validated on host without hardware. + +## Source of Truth + +- Authoritative requirements: + - https://git.mannheim.ccc.de/C3MA/DD3-LoRa-Bridge-MultiSender/src/branch/lora-refactor/Requirements.md +- Pinned C++ baseline for compatibility and fixtures: + - `vendor/dd3-cpp` at commit `a3c61f9b929fbc55bfb502b443fba2f98023b3f1` + +## Workspace Layout + +- `crates/dd3_protocol` + - Pure protocol crate (manual wire codec, no serde for wire) + - CRC16-CCITT, LoRa frame encode/decode, ACK payload codec + - Payload schema v3 encode/decode (sparse 30-slot present mask) + - Chunk reassembly with deterministic reset/error behavior +- `crates/dd3_contracts` + - Pure contracts/formatting crate + - MQTT state JSON, HA discovery topic/payload, CSV formatting + - Device ID sanitize/normalize, HTML escape, URL component encode + - Manufacturer constant lock: `HA_MANUFACTURER = "AcidBurns"` +- `crates/dd3_core` + - Deterministic sender state machine and receiver pipeline + - Trait-based I/O boundaries (`Clock`, `Radio`, `Publisher`, `Storage`, `StatusSink`) +- `crates/dd3_sim` + - Host simulator for deterministic behavior tests + - Fake clock, fake lossy/duplicate/delayed radio, scenario runner and mocks +- `crates/dd3_firmware` + - Embedded integration placeholder crate + - Feature flags for `esp-idf` / `esp-hal`, role-selection scaffold +- `crates/xtask` + - Fixture sync/provenance checks and manufacturer drift guard + +Other folders: + +- `fixtures/` + - Protocol binary fixtures (`frames`, `chunks`, `payload_v3`) + - Contract snapshots (`mqtt_state`, `ha_discovery`, `sd_csv`) +- `docs/` + - `SPEC_LINKS.md`, `TEST_STRATEGY.md`, `INTEROP_CHECKLIST.md` +- `fuzz/` + - Fuzz targets for decode/sanitize entry points + +## What Is Implemented + +### 1) Byte-Exact Protocol Compatibility (`dd3_protocol`) + +Implementation details: + +- Frame format: + - `[msg_kind:1][short_id_be:2][payload][crc16_ccitt_be:2]` +- Message kinds: + - `BatchUp = 0` + - `AckDown = 1` +- ACK payload format: + - Fixed 7 bytes: `[flags:1][batch_id_be:2][epoch_utc_be:4]` + - `flags bit0 = time_valid` +- CRC: + - CRC16-CCITT with init `0xFFFF`, poly `0x1021` +- Payload schema v3: + - 30-slot sparse window with strict `present_mask` and `n` validation + - Supports sync-request packets (`n == 0`, `present_mask == 0`) +- Chunk reassembly: + - Deterministic in-order completion + - Reset on mismatch/out-of-order/invalid total length + +### 2) Contract Stability (`dd3_contracts`) + +Implementation details: + +- Home Assistant discovery contract: + - Topic: `homeassistant/sensor///config` + - `unique_id = _` + - Device metadata includes model `DD3-LoRa-Bridge` and manufacturer `AcidBurns` +- MQTT state JSON: + - Stable key set and semantics (legacy keys intentionally absent) +- CSV output: + - Fixed header: + - `ts_utc,ts_hms_local,p_w,p1_w,p2_w,p3_w,e_kwh,bat_v,bat_pct,rssi,snr,err_m,err_d,err_tx,err_last` + - Stable numeric formatting and optional `snr` blank when NaN +- Sanitization/escaping: + - `sanitize_device_id` accepts `XXXX` or `dd3-XXXX`, normalizes to `dd3-XXXX` + - Rejects traversal/percent/invalid-hex forms + - Includes `html_escape` and URL component encoding + +### 3) Deterministic Sender/Receiver Core (`dd3_core`) + +Sender state machine implementation: + +- Unsynced guardrail: + - Sends only sync-request packets every 15s until valid ACK time bootstrap +- Time bootstrap gate: + - Requires `time_valid=1` and `epoch >= MIN_ACCEPTED_EPOCH_UTC` +- Cadence: + - Sampling at 1 Hz + - Batch cadence at 30s +- Stop-and-wait: + - One inflight batch only + - Retries reuse cached encoded bytes +- ACK handling: + - Adaptive receive windows from airtime + RTT/miss-streak logic + - Bounded retries (`BATCH_MAX_RETRIES`) +- Catch-up and queueing: + - Immediate send when backlog exists and no ACK pending + - Queue depth bounded to `BATCH_QUEUE_DEPTH` +- Fault counters: + - Reset on first valid sync and each UTC hour boundary after sync + +Receiver pipeline implementation: + +- Continuous RX validation pipeline: + - Frame decode -> reassembly -> payload decode -> sparse reconstruction +- Sender identity checks before updates: + - Unknown/mismatched sender is rejected before publish/log updates +- ACK behavior: + - ACK repeats per baseline constants +- Duplicate handling: + - Duplicate batches are ACKed + - Duplicate publish/log is suppressed + - Duplicate counters are updated + +### 4) Simulator (`dd3_sim`) + +Implementation details: + +- `FakeClock` with deterministic time advancement +- `FakeRadioBus` with configurable: + - packet loss + - duplication + - delay +- `ScenarioRunner` and mocks for: + - publisher + - storage + - status sink + +Implemented scenario tests cover: + +- no receiver: sync-requests only +- bootstrap unlock via valid ACK +- packet loss progression without duplicate commit +- ACK mismatch ignored +- backpressure queue bounded +- duplicate batch suppression with counter updates + +### 5) Firmware Placeholder (`dd3_firmware`) + +- Compile-safe scaffold for embedded integration +- Role detection function for GPIO14 exists as placeholder +- Full hardware integration is intentionally deferred + +## Test Strategy and Coverage + +The repo follows four test pillars documented in `docs/TEST_STRATEGY.md`: + +1. Byte-exact protocol compatibility +2. Contract stability +3. Deterministic state-machine behavior +4. Fuzz/property robustness + +Key test files: + +- `crates/dd3_protocol/tests/protocol_tests.rs` +- `crates/dd3_contracts/tests/contracts_tests.rs` +- `crates/dd3_sim/tests/state_machine_tests.rs` + +## Fixtures and Provenance + +- Fixture metadata is tracked in `fixtures/protocol/SOURCES.md` +- Fixture tooling is in `crates/xtask/src/main.rs` + +Commands: + +- `cargo run -p xtask -- sync-fixtures` +- `cargo run -p xtask -- verify-fixture-sources` +- `cargo run -p xtask -- check-manufacturer` + +Note on `full_30` payload fixture: + +- Upstream pinned baseline vector contains a truncated raw byte array. +- This repo stores both: + - `fixtures/protocol/payload_v3/full_30_upstream_raw.bin` (raw upstream) + - `fixtures/protocol/payload_v3/full_30.bin` (canonicalized for codec semantics) + +## Constants Locked to Baseline + +See `docs/SPEC_LINKS.md` for full extraction references. Key locked values include: + +- `MIN_ACCEPTED_EPOCH_UTC = 1769904000` +- `SYNC_REQUEST_INTERVAL_MS = 15000` +- `METER_SAMPLE_INTERVAL_MS = 1000` +- `METER_SEND_INTERVAL_MS = 30000` +- `BATCH_MAX_RETRIES = 2` +- `BATCH_QUEUE_DEPTH = 10` +- `ACK_REPEAT_COUNT = 3` +- `ACK_REPEAT_DELAY_MS = 200` +- `HA_MANUFACTURER = "AcidBurns"` + +## Build and Run + +Prerequisites: + +- Rust stable toolchain (`rustup`, `cargo`, `rustfmt`, `clippy`) +- On Windows MSVC target: Visual C++ Build Tools (`link.exe` available) + +Commands: + +- `make test` +- `make lint` +- `make fuzz-smoke` + +Equivalent cargo commands: + +- `cargo test --workspace` +- `cargo fmt --all -- --check` +- `cargo clippy --workspace --all-targets -- -D warnings` + +## Interoperability Plan + +Use `docs/INTEROP_CHECKLIST.md` for two-way verification: + +- Rust Receiver <-> C++ Sender +- Rust Sender <-> C++ Receiver +- MQTT/CSV contract comparison checks + +## Current Scope and Next Phase + +Current scope is host-first protocol/contracts/core/sim validation. + +Next phase is hardware integration in `dd3_firmware`: + +- real GPIO14 role read (HIGH sender / LOW receiver) +- radio + transport glue on embedded target +- time/local-date partitioning wiring with platform services