C3MA/DD3-LoRa-Bridge-MultiSender
6
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
0577464ec5266300f36e36a9890f0432fc10d8e0
DD3-LoRa-Bridge-MultiSender/Requirements.md
acidburns 0577464ec5 refactor: stabilize legacy-core linking and header ownership 
- Make include/ the canonical declarations for data_model/html_util/json_codec and convert dd3_legacy_core header copies to thin forwarders.
- Add stable public forwarders for app_context/receiver_pipeline/sender_state_machine and update refactor smoke test to stop using ../../src includes.
- Force-link dd3_legacy_core from setup() to ensure deterministic PlatformIO LDF linking across firmware envs.
- Refresh docs (README, Requirements, docs/TESTS.md) to reflect current module paths and smoke-test include strategy.
2026-02-20 23:29:50 +01:00

532 lines
18 KiB
Markdown
Raw Blame History

# Firmware Requirements (Rust Port Preparation)
## 1. Scope
This document defines the behavior that must be preserved when recreating this firmware in another language (target: Rust).
It is based on the current `lora-refactor` code state and captures:
- functional behavior
- protocol/data contracts
- module and function responsibilities
- runtime state-machine requirements
Function names below are C++ references. Rust naming/layout may differ, but the behavior must remain equivalent.
## 2. Refactored Architecture Baseline
The `lora-refactor` branch split role-specific runtime from the previous large `main.cpp` into dedicated modules while keeping a single firmware image:
- `src/main.cpp` is a thin coordinator that:
- detects role and initializes shared platform subsystems,
- prepares role module configuration,
- calls `begin()` once,
- delegates runtime in `loop()`.
- sender runtime ownership:
- `src/sender_state_machine.h`
- `src/sender_state_machine.cpp`
- receiver runtime ownership:
- `src/receiver_pipeline.h`
- `src/receiver_pipeline.cpp`
- receiver shared mutable state used by setup wiring and runtime:
- `src/app_context.h` (`ReceiverSharedState`)
Sender state machine invariants must remain behavior-equivalent:
- single inflight batch at a time,
- ACK acceptance only for matching `batch_id`,
- retry bounded by `BATCH_MAX_RETRIES`,
- queue depth bounded by `BATCH_QUEUE_DEPTH`.
## 3. System-Level Requirements
- Role selection:
- `Sender` when `GPIO14` reads HIGH.
- `Receiver` when `GPIO14` reads LOW.
- Device identity:
- derive `short_id` from MAC bytes 4/5.
- canonical `device_id` format: `dd3-XXXX` uppercase hex.
- LoRa transport:
- frame format: `[msg_kind][short_id_be][payload][crc16_ccitt]`.
- reject invalid CRC/msg-kind/length.
- Payload codec:
- schema `3` with `present_mask` (30-bit sparse second map).
- support `n==0` sync-request packets.
- Time bootstrap guardrail:
- sender must not run normal sampling/transmit until valid ACK time received.
- accept ACK time only if `time_valid=1` and `epoch >= MIN_ACCEPTED_EPOCH_UTC`.
- sender fault counters reset when first valid sync is accepted.
- after first sync, sender fault counters reset again at each UTC hour boundary.
- Sampling/transmit cadence:
- sender sample cadence 1 Hz.
- sender batch cadence 30 s.
- when sender backlog exists (`batch_count > 1`) and no ACK is pending, sender performs immediate catch-up sends (still stop-and-wait with one inflight batch).
- sync-request cadence 15 s while unsynced.
- sender retransmits reuse cached encoded payload bytes for same inflight batch.
- sender ACK receive window is adaptive from airtime + observed ACK RTT, with expanded second window on miss.
- Receiver behavior:
- decode/reconstruct sparse timestamps.
- ACK accepted batches promptly.
- reject unknown/mismatched sender identities before ACK and before SD/MQTT/web updates.
- update MQTT, web status, SD logging.
- Persistence:
- Wi-Fi/MQTT/NTP/web credentials in Preferences namespace `dd3cfg`.
- Web auth defaults:
- `WEB_AUTH_REQUIRE_STA=true`
- `WEB_AUTH_REQUIRE_AP=true`
- Web and display time rendering:
- local timezone from `TIMEZONE_TZ`.
- Sender diagnostics:
- structured sender diagnostics are emitted to serial debug output only.
- diagnostics do not change LoRa payload schema or remap payload fields.
- SD logging:
- CSV columns include both `ts_utc` and `ts_hms_local`.
- per-day CSV file partitioning uses local date (`TIMEZONE_TZ`) under `/dd3/<device_id>/YYYY-MM-DD.csv`.
- history day-file resolution prefers local-date filenames and falls back to legacy UTC-date filenames.
- history parser supports both current (`ts_utc,ts_hms_local,p_w,...`) and legacy (`ts_utc,p_w,...`) layouts.
## 4. Protocol and Data Contracts
- `LoraMsgKind`:
- `BatchUp=0`
- `AckDown=1`
- `AckDown` payload fixed length `7` bytes:
- `[flags:1][batch_id_be:2][epoch_utc_be:4]`
- `flags bit0 = time_valid`
- sender acceptance window is implementation-adaptive; payload format stays unchanged.
- `BatchInput`:
- fixed arrays length `30` (`energy_wh`, `p1_w`, `p2_w`, `p3_w`)
- `present_mask` must satisfy: only low 30 bits used and `bit_count == n`
- Timestamp constraints:
- receiver rejects decoded data whose timestamps are below `MIN_ACCEPTED_EPOCH_UTC`
- CSV header (current required layout):
- `ts_utc,ts_hms_local,p_w,p1_w,p2_w,p3_w,e_kwh,bat_v,bat_pct,rssi,snr,err_m,err_d,err_tx,err_last`
- Home Assistant discovery contract:
- topic: `homeassistant/sensor/<device_id>/<key>/config`
- `unique_id`: `<device_id>_<key>`
- `device.identifiers`: `["<device_id>"]`
- `device.name`: `<device_id>`
- `device.model`: `DD3-LoRa-Bridge`
- `device.manufacturer`: `AcidBurns`
- drift guards:
- canonical value is `HA_MANUFACTURER` in `include/config.h`,
- compile-time lock via `static_assert` in `include/config.h`,
- script guard `test/check_ha_manufacturer.ps1`,
- smoke test guard `test/test_refactor_smoke/test_refactor_smoke.cpp`.
## 5. Module and Function Requirements
## `src/config.cpp`
- `DeviceRole detect_role()`
- configure role pin input pulldown and map to sender/receiver role.
## `lib/dd3_legacy_core/src/data_model.cpp`
- `void init_device_ids(uint16_t&, char*, size_t)`
- read MAC, derive short ID, format canonical device ID.
- `const char *rx_reject_reason_text(RxRejectReason)`
- stable mapping for diagnostics and payloads.
## `lib/dd3_legacy_core/src/html_util.cpp`
- `String html_escape(const String&)`
- escape `& < > " '`.
- `String url_encode_component(const String&)`
- percent-encode non-safe characters.
- `bool sanitize_device_id(const String&, String&)`
- accept `XXXX` or `dd3-XXXX`; reject path traversal, `%`, invalid hex.
- Internal helpers to preserve behavior:
- `is_hex_char`
- `to_upper_hex4`
## `src/meter_driver.cpp`
- `void meter_init()`
- configure `Serial2` at `9600 7E1`, RX pin `PIN_METER_RX`, RX buffer size `8192` on ESP32.
- `bool meter_poll_frame(const char *&, size_t&)`
- incremental frame collector with start `/`, end `!`, timeout, overflow handling.
- `bool meter_parse_frame(const char*, size_t, MeterData&)`
- parse OBIS values and set meter data fields.
- `bool meter_read(MeterData&)`
- compatibility wrapper around poll+parse.
- `void meter_get_stats(MeterDriverStats&)`
- expose parser/UART counters for sender-local diagnostics.
- Internal parse helpers to preserve numeric behavior:
- `detect_obis_field`
- `parse_decimal_fixed`
- `parse_obis_ascii_payload_value`
- `parse_obis_ascii_unit_scale`
- `hex_nibble`
- `parse_obis_hex_payload_u32`
- `meter_debug_log`
## `src/power_manager.cpp`
- `void power_sender_init()`
- sender low-power setup (CPU freq, Wi-Fi/BT off, ADC setup).
- `void power_receiver_init()`
- receiver power setup.
- `void power_configure_unused_pins_sender()`
- configure known-unused pins with pulldown.
- `void read_battery(MeterData&)`
- averaged ADC conversion and voltage calibration.
- `uint8_t battery_percent_from_voltage(float)`
- LUT + interpolation.
- `void light_sleep_ms(uint32_t)`
- timer-based light sleep.
- `void go_to_deep_sleep(uint32_t)`
- timer-based deep sleep.
## `src/time_manager.cpp`
- `void time_receiver_init(const char*, const char*)`
- configure NTP servers and timezone env.
- `uint32_t time_get_utc()`
- return epoch or `0` when not plausible.
- updates "clock plausible" state independently from sync state.
- `bool time_is_synced()`
- true only after explicit sync signals (NTP callback/status or trusted `time_set_utc`).
- `void time_set_utc(uint32_t)`
- set system time and sync flags.
- `void time_get_local_hhmm(char*, size_t)`
- timezone-based local `HH:MM` output.
- `uint32_t time_get_last_sync_utc()`
- `uint32_t time_get_last_sync_age_sec()`
- Internal behavior-critical helpers:
- `note_last_sync`
- `mark_synced`
- `ntp_sync_notification_cb`
- `ensure_timezone_set`
## `src/lora_transport.cpp`
- `void lora_init()`
- initialize SX1276 with configured LoRa params.
- `bool lora_send(const LoraPacket&)`
- frame pack + CRC append + transmit.
- `bool lora_receive(LoraPacket&, uint32_t timeout_ms)`
- parse frame, validate, return metadata including RSSI/SNR.
- `RxRejectReason lora_get_last_rx_reject_reason()`
- consume-and-clear reject reason.
- `bool lora_get_last_rx_signal(int16_t&, float&)`
- access last RX signal snapshot.
- `void lora_idle()`
- `void lora_sleep()`
- `void lora_receive_continuous()`
- `bool lora_receive_window(LoraPacket&, uint32_t)`
- `uint32_t lora_airtime_ms(size_t)`
- compute packet airtime from SF/BW/CR/preamble.
- Internal behavior-critical helpers:
- `note_reject`
- `crc16_ccitt`
## `lib/dd3_legacy_core/src/payload_codec.cpp`
- `bool encode_batch(const BatchInput&, uint8_t*, size_t, size_t*)`
- schema v3 encoder with metadata, sparse present mask, delta coding.
- `bool decode_batch(const uint8_t*, size_t, BatchInput*)`
- strict schema/magic/flags decode + bounds checks.
- Varint primitives:
- `uleb128_encode`, `uleb128_decode`
- `zigzag32`, `unzigzag32`
- `svarint_encode`, `svarint_decode`
- Internal helpers:
- `write_u16_le`, `write_u32_le`
- `read_u16_le`, `read_u32_le`
- `ensure_capacity`
- `bit_count32`
- Optional self-test:
- `payload_codec_self_test` (when `PAYLOAD_CODEC_TEST`).
## `lib/dd3_legacy_core/src/json_codec.cpp`
- `bool meterDataToJson(const MeterData&, String&)`
- create MQTT state JSON with stable field semantics.
- Internal numeric formatting helpers:
- `round2`
- `round_to_i32`
- `short_id_from_device_id`
- `format_float_2`
- `set_int_or_null`
## `src/mqtt_client.cpp`
- `void mqtt_init(const WifiMqttConfig&, const char*)`
- `void mqtt_loop()`
- `bool mqtt_is_connected()`
- `bool mqtt_publish_state(const MeterData&)`
- `bool mqtt_publish_faults(const char*, const FaultCounters&, FaultType, uint32_t)`
- `bool mqtt_publish_discovery(const char*)`
- `bool mqtt_publish_test(const char*, const String&)` (test mode only)
- Internal behavior-critical helpers:
- `fault_text`
- `mqtt_connect`
- `publish_discovery_sensor`
- discovery payload uses canonical device identity fields and `manufacturer=AcidBurns`
## `src/wifi_manager.cpp`
- `void wifi_manager_init()`
- `bool wifi_load_config(WifiMqttConfig&)`
- `bool wifi_save_config(const WifiMqttConfig&)`
- returns `false` when any Preferences write/verify fails.
- `bool wifi_connect_sta(const WifiMqttConfig&, uint32_t timeout_ms)`
- `void wifi_start_ap(const char*, const char*)`
- `bool wifi_is_connected()`
- `String wifi_get_ssid()`
## `src/sd_logger.cpp`
- `void sd_logger_init()`
- `bool sd_logger_is_ready()`
- `void sd_logger_log_sample(const MeterData&, bool include_error_text)`
- append/create per-day CSV under `/dd3/<device_id>/YYYY-MM-DD.csv` using local calendar date from `TIMEZONE_TZ`.
- Internal behavior-critical helpers:
- `fault_text`
- `ensure_dir`
- `format_date_local`
- `format_hms_local`
## `src/display_ui.cpp`
Public display API that must remain behavior-equivalent:
- `display_power_down`
- `display_init`
- `display_set_role`
- `display_set_self_ids`
- `display_set_sender_statuses`
- `display_set_last_meter`
- `display_set_last_read`
- `display_set_last_tx`
- `display_set_sender_queue`
- `display_set_sender_batches`
- `display_set_last_error`
- `display_set_receiver_status`
- `display_set_test_code` (test mode)
- `display_set_test_code_for_sender` (test mode)
- `display_tick`
Internal rendering helpers to preserve behavior:
- `oled_set_power`
- `age_seconds`
- `round_power_w`
- `render_last_error_line`
- `render_last_sync_line`
- `render_sender_status`
- `render_sender_measurement`
- `render_receiver_status`
- `render_receiver_sender`
## `src/web_server.cpp`
Public web API:
- `web_server_set_config`
- `web_server_set_sender_faults`
- `web_server_set_last_batch`
- `web_server_begin_ap`
- `web_server_begin_sta`
- `web_server_loop`
Internal route/state functions to preserve behavior:
- `format_local_hms`
- `format_epoch_local_hms`
- `timestamp_age_seconds`
- `round_power_w`
- `auth_required`
- `fault_text`
- `ensure_auth`
- `html_header`
- `html_footer`
- `format_faults`
- `sanitize_sd_download_path`
- `checkbox_checked`
- `sanitize_history_device_id`
- `sanitize_download_filename`
- `history_reset`
- `history_date_from_epoch_local`
- `history_date_from_epoch_utc` (legacy fallback mapping)
- `history_open_next_file`
- `history_parse_line`
- `history_tick`
- `render_sender_block`
- `append_sd_listing`
- `handle_root`
- `handle_wifi_get`
- `handle_wifi_post`
- `handle_sender`
- `handle_manual`
- `handle_history_start`
- `handle_history_data`
- `handle_sd_download`
## `src/test_mode.cpp` (`ENABLE_TEST_MODE`)
- `test_sender_loop`
- periodic JSON test frame transmit.
- `test_receiver_loop`
- decode test JSON, update display test markers, publish MQTT test topic.
## `src/app_context.h`
- `ReceiverSharedState`
- retains receiver-owned shared status/fault/discovery state used by setup wiring and runtime.
## `src/sender_state_machine.h/.cpp` (Sender Runtime)
Public API:
- `SenderStateMachineConfig`
- `SenderStats`
- `SenderStateMachine::begin(...)`
- `SenderStateMachine::loop()`
- `SenderStateMachine::stats()`
Behavior-critical internals (migrated from pre-refactor `main.cpp`) that must remain equivalent:
- Logging/utilities:
- `serial_debug_printf`
- `bit_count32`
- `abs_diff_u32`
- Meter-time anchoring and ingest:
- `meter_time_update_snapshot`
- `set_last_meter_sample`
- `parse_meter_frame_sample`
- `meter_queue_push_latest`
- `meter_reader_task_entry`
- `meter_reader_start`
- `meter_reader_pump`
- Sender state/data handling:
- `update_battery_cache`
- `battery_sample_due`
- `batch_queue_drop_oldest`
- `sender_note_rx_reject`
- `sender_log_diagnostics`
- `batch_queue_peek`
- `batch_queue_enqueue`
- `reset_build_counters`
- `append_meter_sample`
- `last_sample_ts`
- Sender fault handling:
- `note_fault`
- `clear_faults`
- `sender_reset_fault_stats`
- `sender_reset_fault_stats_on_first_sync`
- `sender_reset_fault_stats_on_hour_boundary`
- `age_seconds`
- `counters_changed`
- `publish_faults_if_needed`
- Sender-specific encoding/scheduling:
- `kwh_to_wh_from_float`
- `float_to_i16_w`
- `float_to_i16_w_clamped`
- `battery_mv_from_voltage`
- `compute_batch_ack_timeout_ms`
- `send_batch_payload`
- `invalidate_inflight_encode_cache`
- `prepare_inflight_from_queue`
- `send_inflight_batch`
- `send_meter_batch`
- `send_sync_request`
- `resend_inflight_batch`
- `finish_inflight_batch`
- `sender_loop`
## `src/receiver_pipeline.h/.cpp` (Receiver Runtime)
Public API:
- `ReceiverPipelineConfig`
- `ReceiverStats`
- `ReceiverPipeline::begin(...)`
- `ReceiverPipeline::loop()`
- `ReceiverPipeline::stats()`
Behavior-critical internals (migrated from pre-refactor `main.cpp`) that must remain equivalent:
- Receiver setup/state:
- `init_sender_statuses`
- Fault handling/publish:
- `note_fault`
- `clear_faults`
- `age_seconds`
- `counters_changed`
- `publish_faults_if_needed`
- Binary helpers and ID conversion:
- `write_u16_le`
- `read_u16_le`
- `write_u16_be`
- `read_u16_be`
- `write_u32_be`
- `read_u32_be`
- `sender_id_from_short_id`
- `short_id_from_sender_id`
- LoRa RX/TX pipeline:
- `compute_batch_rx_timeout_ms`
- `send_batch_ack`
- `reset_batch_rx`
- `process_batch_packet`
- `receiver_loop`
## `src/main.cpp` (Thin Coordinator)
Current core orchestration requirements:
- `setup`
- initialize shared subsystems once,
- force-link `dd3_legacy_core` before first legacy-core symbol use (`dd3_legacy_core_force_link()`),
- instantiate role config and call role `begin`,
- keep role-specific runtime out of this file.
- `loop`
- delegate to `SenderStateMachine::loop()` or `ReceiverPipeline::loop()` by role.
- Watchdog wrapper remains in coordinator:
- `watchdog_init`
- `watchdog_kick`
## 6. Rust Porting Constraints and Recommendations
- Preserve wire compatibility first:
- LoRa frame byte layout, CRC16, ACK format, payload schema v3.
- sender optimization changes must not alter payload field meanings.
- Preserve persistent storage keys:
- Preferences keys (`ssid`, `pass`, `mqhost`, `mqport`, `mquser`, `mqpass`, `ntp1`, `ntp2`, `webuser`, `webpass`, `valid`).
- Preserve timing constants and acceptance thresholds:
- bootstrap guardrail, retry counts, schedule intervals, min accepted epoch.
- Preserve CSV output layout exactly:
- consumers (history parser and external tooling) depend on it.
- preserve reader compatibility for both current and legacy layouts.
- Preserve enum meanings:
- `FaultType`, `RxRejectReason`, `LoraMsgKind`.
Suggested Rust module split:
- `config`, `ids`, `meter`, `power`, `time`, `lora_transport`, `payload_codec`, `sender_state_machine`, `receiver_pipeline`, `app_context`, `mqtt`, `wifi_cfg`, `sd_log`, `web`, `display`, `runtime`.
Suggested Rust primitives:
- async task for meter reader + bounded channel (drop-oldest behavior).
- explicit state structs for sender/receiver loops.
- serde-free/manual codec for wire compatibility where needed.
## 7. Port Validation Checklist
- Sender unsynced boot sends only sync requests.
- ACK time bootstrap unlocks normal sender sampling.
- Sparse present-mask encode/decode round-trip matches C++.
- Receiver reconstructs timestamps correctly for gaps.
- Duplicate batch handling updates counters and suppresses duplicate publish/log.
- Web UI shows `epoch (HH:MM:SS TZ)` local time.
- SD CSV header/fields match expected order.
- SD daily files roll over at local midnight (`TIMEZONE_TZ`), not UTC midnight.
- History endpoint reads current and legacy CSV layouts successfully.
- History endpoint can read both local-date and legacy UTC-date day filenames.
- MQTT state/fault payload fields match existing names and semantics.
## 8. Port Readiness Audit (2026-02-20)
Evidence checked on `lora-refactor`:
- build verification:
- `pio run -e lilygo-t3-v1-6-1`
- `pio run -e lilygo-t3-v1-6-1-test`
- drift guard verification:
- `powershell -ExecutionPolicy Bypass -File test/check_ha_manufacturer.ps1`
- refactor ownership verification:
- sender state machine state/API present in `src/sender_state_machine.h/.cpp`,
- receiver pipeline API present in `src/receiver_pipeline.h/.cpp`,
- coordinator remains thin in `src/main.cpp`.
Findings:
- Requirements are functionally met by current C++ baseline from static/code-build checks.
- The old requirement ownership under `src/main.cpp` was stale; this document now maps that behavior to `sender_state_machine` and `receiver_pipeline`.
- No wire/protocol or persistence contract drift found in this audit.
Reference in New Issue View Git Blame Copy Permalink