Add comprehensive README for workspace functionality and implementation
This commit is contained in:
237
README.md
Normal file
237
README.md
Normal file
@@ -0,0 +1,237 @@
|
|||||||
|
# DD3 LoRa Bridge Rust Port (Host-First, Test-First)
|
||||||
|
|
||||||
|
This repository is a Rust workspace port of the DD3 LoRa Bridge with compatibility as the primary goal.
|
||||||
|
|
||||||
|
The implementation is organized so protocol, contracts, and state-machine behavior can be validated on host without hardware.
|
||||||
|
|
||||||
|
## Source of Truth
|
||||||
|
|
||||||
|
- Authoritative requirements:
|
||||||
|
- https://git.mannheim.ccc.de/C3MA/DD3-LoRa-Bridge-MultiSender/src/branch/lora-refactor/Requirements.md
|
||||||
|
- Pinned C++ baseline for compatibility and fixtures:
|
||||||
|
- `vendor/dd3-cpp` at commit `a3c61f9b929fbc55bfb502b443fba2f98023b3f1`
|
||||||
|
|
||||||
|
## Workspace Layout
|
||||||
|
|
||||||
|
- `crates/dd3_protocol`
|
||||||
|
- Pure protocol crate (manual wire codec, no serde for wire)
|
||||||
|
- CRC16-CCITT, LoRa frame encode/decode, ACK payload codec
|
||||||
|
- Payload schema v3 encode/decode (sparse 30-slot present mask)
|
||||||
|
- Chunk reassembly with deterministic reset/error behavior
|
||||||
|
- `crates/dd3_contracts`
|
||||||
|
- Pure contracts/formatting crate
|
||||||
|
- MQTT state JSON, HA discovery topic/payload, CSV formatting
|
||||||
|
- Device ID sanitize/normalize, HTML escape, URL component encode
|
||||||
|
- Manufacturer constant lock: `HA_MANUFACTURER = "AcidBurns"`
|
||||||
|
- `crates/dd3_core`
|
||||||
|
- Deterministic sender state machine and receiver pipeline
|
||||||
|
- Trait-based I/O boundaries (`Clock`, `Radio`, `Publisher`, `Storage`, `StatusSink`)
|
||||||
|
- `crates/dd3_sim`
|
||||||
|
- Host simulator for deterministic behavior tests
|
||||||
|
- Fake clock, fake lossy/duplicate/delayed radio, scenario runner and mocks
|
||||||
|
- `crates/dd3_firmware`
|
||||||
|
- Embedded integration placeholder crate
|
||||||
|
- Feature flags for `esp-idf` / `esp-hal`, role-selection scaffold
|
||||||
|
- `crates/xtask`
|
||||||
|
- Fixture sync/provenance checks and manufacturer drift guard
|
||||||
|
|
||||||
|
Other folders:
|
||||||
|
|
||||||
|
- `fixtures/`
|
||||||
|
- Protocol binary fixtures (`frames`, `chunks`, `payload_v3`)
|
||||||
|
- Contract snapshots (`mqtt_state`, `ha_discovery`, `sd_csv`)
|
||||||
|
- `docs/`
|
||||||
|
- `SPEC_LINKS.md`, `TEST_STRATEGY.md`, `INTEROP_CHECKLIST.md`
|
||||||
|
- `fuzz/`
|
||||||
|
- Fuzz targets for decode/sanitize entry points
|
||||||
|
|
||||||
|
## What Is Implemented
|
||||||
|
|
||||||
|
### 1) Byte-Exact Protocol Compatibility (`dd3_protocol`)
|
||||||
|
|
||||||
|
Implementation details:
|
||||||
|
|
||||||
|
- Frame format:
|
||||||
|
- `[msg_kind:1][short_id_be:2][payload][crc16_ccitt_be:2]`
|
||||||
|
- Message kinds:
|
||||||
|
- `BatchUp = 0`
|
||||||
|
- `AckDown = 1`
|
||||||
|
- ACK payload format:
|
||||||
|
- Fixed 7 bytes: `[flags:1][batch_id_be:2][epoch_utc_be:4]`
|
||||||
|
- `flags bit0 = time_valid`
|
||||||
|
- CRC:
|
||||||
|
- CRC16-CCITT with init `0xFFFF`, poly `0x1021`
|
||||||
|
- Payload schema v3:
|
||||||
|
- 30-slot sparse window with strict `present_mask` and `n` validation
|
||||||
|
- Supports sync-request packets (`n == 0`, `present_mask == 0`)
|
||||||
|
- Chunk reassembly:
|
||||||
|
- Deterministic in-order completion
|
||||||
|
- Reset on mismatch/out-of-order/invalid total length
|
||||||
|
|
||||||
|
### 2) Contract Stability (`dd3_contracts`)
|
||||||
|
|
||||||
|
Implementation details:
|
||||||
|
|
||||||
|
- Home Assistant discovery contract:
|
||||||
|
- Topic: `homeassistant/sensor/<device_id>/<key>/config`
|
||||||
|
- `unique_id = <device_id>_<key>`
|
||||||
|
- Device metadata includes model `DD3-LoRa-Bridge` and manufacturer `AcidBurns`
|
||||||
|
- MQTT state JSON:
|
||||||
|
- Stable key set and semantics (legacy keys intentionally absent)
|
||||||
|
- CSV output:
|
||||||
|
- Fixed header:
|
||||||
|
- `ts_utc,ts_hms_local,p_w,p1_w,p2_w,p3_w,e_kwh,bat_v,bat_pct,rssi,snr,err_m,err_d,err_tx,err_last`
|
||||||
|
- Stable numeric formatting and optional `snr` blank when NaN
|
||||||
|
- Sanitization/escaping:
|
||||||
|
- `sanitize_device_id` accepts `XXXX` or `dd3-XXXX`, normalizes to `dd3-XXXX`
|
||||||
|
- Rejects traversal/percent/invalid-hex forms
|
||||||
|
- Includes `html_escape` and URL component encoding
|
||||||
|
|
||||||
|
### 3) Deterministic Sender/Receiver Core (`dd3_core`)
|
||||||
|
|
||||||
|
Sender state machine implementation:
|
||||||
|
|
||||||
|
- Unsynced guardrail:
|
||||||
|
- Sends only sync-request packets every 15s until valid ACK time bootstrap
|
||||||
|
- Time bootstrap gate:
|
||||||
|
- Requires `time_valid=1` and `epoch >= MIN_ACCEPTED_EPOCH_UTC`
|
||||||
|
- Cadence:
|
||||||
|
- Sampling at 1 Hz
|
||||||
|
- Batch cadence at 30s
|
||||||
|
- Stop-and-wait:
|
||||||
|
- One inflight batch only
|
||||||
|
- Retries reuse cached encoded bytes
|
||||||
|
- ACK handling:
|
||||||
|
- Adaptive receive windows from airtime + RTT/miss-streak logic
|
||||||
|
- Bounded retries (`BATCH_MAX_RETRIES`)
|
||||||
|
- Catch-up and queueing:
|
||||||
|
- Immediate send when backlog exists and no ACK pending
|
||||||
|
- Queue depth bounded to `BATCH_QUEUE_DEPTH`
|
||||||
|
- Fault counters:
|
||||||
|
- Reset on first valid sync and each UTC hour boundary after sync
|
||||||
|
|
||||||
|
Receiver pipeline implementation:
|
||||||
|
|
||||||
|
- Continuous RX validation pipeline:
|
||||||
|
- Frame decode -> reassembly -> payload decode -> sparse reconstruction
|
||||||
|
- Sender identity checks before updates:
|
||||||
|
- Unknown/mismatched sender is rejected before publish/log updates
|
||||||
|
- ACK behavior:
|
||||||
|
- ACK repeats per baseline constants
|
||||||
|
- Duplicate handling:
|
||||||
|
- Duplicate batches are ACKed
|
||||||
|
- Duplicate publish/log is suppressed
|
||||||
|
- Duplicate counters are updated
|
||||||
|
|
||||||
|
### 4) Simulator (`dd3_sim`)
|
||||||
|
|
||||||
|
Implementation details:
|
||||||
|
|
||||||
|
- `FakeClock` with deterministic time advancement
|
||||||
|
- `FakeRadioBus` with configurable:
|
||||||
|
- packet loss
|
||||||
|
- duplication
|
||||||
|
- delay
|
||||||
|
- `ScenarioRunner` and mocks for:
|
||||||
|
- publisher
|
||||||
|
- storage
|
||||||
|
- status sink
|
||||||
|
|
||||||
|
Implemented scenario tests cover:
|
||||||
|
|
||||||
|
- no receiver: sync-requests only
|
||||||
|
- bootstrap unlock via valid ACK
|
||||||
|
- packet loss progression without duplicate commit
|
||||||
|
- ACK mismatch ignored
|
||||||
|
- backpressure queue bounded
|
||||||
|
- duplicate batch suppression with counter updates
|
||||||
|
|
||||||
|
### 5) Firmware Placeholder (`dd3_firmware`)
|
||||||
|
|
||||||
|
- Compile-safe scaffold for embedded integration
|
||||||
|
- Role detection function for GPIO14 exists as placeholder
|
||||||
|
- Full hardware integration is intentionally deferred
|
||||||
|
|
||||||
|
## Test Strategy and Coverage
|
||||||
|
|
||||||
|
The repo follows four test pillars documented in `docs/TEST_STRATEGY.md`:
|
||||||
|
|
||||||
|
1. Byte-exact protocol compatibility
|
||||||
|
2. Contract stability
|
||||||
|
3. Deterministic state-machine behavior
|
||||||
|
4. Fuzz/property robustness
|
||||||
|
|
||||||
|
Key test files:
|
||||||
|
|
||||||
|
- `crates/dd3_protocol/tests/protocol_tests.rs`
|
||||||
|
- `crates/dd3_contracts/tests/contracts_tests.rs`
|
||||||
|
- `crates/dd3_sim/tests/state_machine_tests.rs`
|
||||||
|
|
||||||
|
## Fixtures and Provenance
|
||||||
|
|
||||||
|
- Fixture metadata is tracked in `fixtures/protocol/SOURCES.md`
|
||||||
|
- Fixture tooling is in `crates/xtask/src/main.rs`
|
||||||
|
|
||||||
|
Commands:
|
||||||
|
|
||||||
|
- `cargo run -p xtask -- sync-fixtures`
|
||||||
|
- `cargo run -p xtask -- verify-fixture-sources`
|
||||||
|
- `cargo run -p xtask -- check-manufacturer`
|
||||||
|
|
||||||
|
Note on `full_30` payload fixture:
|
||||||
|
|
||||||
|
- Upstream pinned baseline vector contains a truncated raw byte array.
|
||||||
|
- This repo stores both:
|
||||||
|
- `fixtures/protocol/payload_v3/full_30_upstream_raw.bin` (raw upstream)
|
||||||
|
- `fixtures/protocol/payload_v3/full_30.bin` (canonicalized for codec semantics)
|
||||||
|
|
||||||
|
## Constants Locked to Baseline
|
||||||
|
|
||||||
|
See `docs/SPEC_LINKS.md` for full extraction references. Key locked values include:
|
||||||
|
|
||||||
|
- `MIN_ACCEPTED_EPOCH_UTC = 1769904000`
|
||||||
|
- `SYNC_REQUEST_INTERVAL_MS = 15000`
|
||||||
|
- `METER_SAMPLE_INTERVAL_MS = 1000`
|
||||||
|
- `METER_SEND_INTERVAL_MS = 30000`
|
||||||
|
- `BATCH_MAX_RETRIES = 2`
|
||||||
|
- `BATCH_QUEUE_DEPTH = 10`
|
||||||
|
- `ACK_REPEAT_COUNT = 3`
|
||||||
|
- `ACK_REPEAT_DELAY_MS = 200`
|
||||||
|
- `HA_MANUFACTURER = "AcidBurns"`
|
||||||
|
|
||||||
|
## Build and Run
|
||||||
|
|
||||||
|
Prerequisites:
|
||||||
|
|
||||||
|
- Rust stable toolchain (`rustup`, `cargo`, `rustfmt`, `clippy`)
|
||||||
|
- On Windows MSVC target: Visual C++ Build Tools (`link.exe` available)
|
||||||
|
|
||||||
|
Commands:
|
||||||
|
|
||||||
|
- `make test`
|
||||||
|
- `make lint`
|
||||||
|
- `make fuzz-smoke`
|
||||||
|
|
||||||
|
Equivalent cargo commands:
|
||||||
|
|
||||||
|
- `cargo test --workspace`
|
||||||
|
- `cargo fmt --all -- --check`
|
||||||
|
- `cargo clippy --workspace --all-targets -- -D warnings`
|
||||||
|
|
||||||
|
## Interoperability Plan
|
||||||
|
|
||||||
|
Use `docs/INTEROP_CHECKLIST.md` for two-way verification:
|
||||||
|
|
||||||
|
- Rust Receiver <-> C++ Sender
|
||||||
|
- Rust Sender <-> C++ Receiver
|
||||||
|
- MQTT/CSV contract comparison checks
|
||||||
|
|
||||||
|
## Current Scope and Next Phase
|
||||||
|
|
||||||
|
Current scope is host-first protocol/contracts/core/sim validation.
|
||||||
|
|
||||||
|
Next phase is hardware integration in `dd3_firmware`:
|
||||||
|
|
||||||
|
- real GPIO14 role read (HIGH sender / LOW receiver)
|
||||||
|
- radio + transport glue on embedded target
|
||||||
|
- time/local-date partitioning wiring with platform services
|
||||||
Reference in New Issue
Block a user