Commit Graph

101 Commits

Author SHA1 Message Date
664ff1d744 sec(sender): ACK rate-limiting, unknown device-ID rejection, fuzz tests
- Add 500 ms minimum interval between accepted ACKs to mitigate replay floods
- Reject ACK packets from unrecognised device IDs (DeviceIdMismatch)
- Add test/test_security_fuzz: negative/boundary tests for decode_batch,
  uleb128_decode, svarint_decode, lora_parse_frame entry points
2026-03-17 12:31:36 +01:00
b9591ce9bb feat(power): 1Hz chunked light-sleep; meter backoff; log throttling
- Replace delay() with light_sleep_chunked_ms() in sender idle path
  (100ms chunks preserve UART FIFO safety at 9600 baud)
- Add ENABLE_LIGHT_SLEEP_IDLE build flag (default: on, fallback: =0)
- Meter reader task: exponential backoff on consecutive poll failures
  (METER_FAIL_BACKOFF_BASE_MS..MAX_MS) to reduce idle Core-0 wakeups
- Configurable SENDER_DIAG_LOG_INTERVAL_MS (5s debug / 30s prod)
- Configurable METER_FRAME_TIMEOUT_CFG_MS, SENDER_CPU_MHZ
- New PlatformIO envs: lowpower, 868-lowpower, lowpower-debug
- Add docs/POWER_OPTIMIZATION.md with measurement plan and Go/No-Go
2026-03-16 16:32:49 +01:00
e89aee7048 Add WiFi reconnection retry logic to recover from unreliable WiFi
- Implement periodic WiFi reconnection attempts when stuck in AP mode
- Add WIFI_RECONNECT_INTERVAL_MS config (default 60s) for configurable retry frequency
- Prevent data loss by automatically attempting to switch back to STA mode
- Maintain AP mode for manual configuration if reconnection fails
- Track WiFi config and last reconnection attempt time in shared state
- Add wifi_try_reconnect_sta() and wifi_restore_ap_mode() helper functions
- Log reconnection attempts and results for debugging
2026-03-11 20:32:15 +01:00
7fbaf77806 Merge remote-tracking branch 'origin/lora-refactor' into lora-refactor
Resolve merge conflict in src/main.cpp by accepting the refactored version
from origin. The remote branch includes significant architectural improvements:

- New app context and state machine structures
- Refactored receiver and sender logic
- Library reorganization (dd3_legacy_core, dd3_transport_logic)
- Test framework enhancements
- Code quality improvements

The local WiFi reconnection feature (commit 32cd065) will be
re-integrated if needed in the new architecture.
2026-03-11 17:05:39 +01:00
32cd0652c9 Improve reliability and add data recovery tools
## Bug Fixes
- Fix integer overflow potential in history bin allocation (web_server.cpp)
  Using uint64_t for intermediate multiplication prevents overflow with different constants

- Prevent data loss during WiFi failures (main.cpp)
  Device now automatically attempts WiFi reconnection every 30 seconds when in AP mode
  Exits AP mode and resumes MQTT transmission as soon as WiFi becomes available
  Data collection and SD logging continue regardless of connectivity

## New Features
- Add standalone MQTT data republisher for lost data recovery
  - Command-line tool (republish_mqtt.py) with interactive and scripting modes
  - GUI tool (republish_mqtt_gui.py) for user-friendly recovery
  - Rate-limited publishing (5 msg/sec default, configurable 1-100)
  - Manual time range selection or auto-detect missing data via InfluxDB
  - Cross-platform support (Windows, macOS, Linux)
  - Converts SD card CSV exports back to MQTT format

## Documentation
- Add comprehensive code review (CODE_REVIEW.md)
  - 16 detailed security and quality assessments
  - Identifies critical HTTPS/auth gaps, medium-priority overflow issues
  - Confirms absence of buffer overflows and unsafe string functions
  - Grade: B+ with areas for improvement

- Add republisher documentation (REPUBLISH_README.md, REPUBLISH_GUI_README.md)
  - Installation and usage instructions
  - Example commands and scenarios
  - Troubleshooting guide
  - Performance characteristics

## Dependencies
- Add requirements_republish.txt
  - paho-mqtt>=1.6.1
  - influxdb-client>=1.18.0

## Impact
- Eliminates data loss scenario where unreliable WiFi leaves device stuck in AP mode
- Provides recovery mechanism for any historical data missed during outages
- Improves code safety with explicit overflow-resistant arithmetic
- Increases operational visibility with comprehensive code review
2026-03-11 17:01:22 +01:00
0577464ec5 refactor: stabilize legacy-core linking and header ownership
- Make include/ the canonical declarations for data_model/html_util/json_codec and convert dd3_legacy_core header copies to thin forwarders.
- Add stable public forwarders for app_context/receiver_pipeline/sender_state_machine and update refactor smoke test to stop using ../../src includes.
- Force-link dd3_legacy_core from setup() to ensure deterministic PlatformIO LDF linking across firmware envs.
- Refresh docs (README, Requirements, docs/TESTS.md) to reflect current module paths and smoke-test include strategy.
2026-02-20 23:29:50 +01:00
b8e0733a89 test: add json stability and discovery payload coverage 2026-02-20 21:32:35 +01:00
ca2cd1880a test: add lora frame and chunk reassembly logic suite 2026-02-20 21:26:51 +01:00
cef1d184ed test: add payload codec regression suite 2026-02-20 21:22:10 +01:00
6acb588069 refactor: move html_util into legacy core library 2026-02-20 21:17:48 +01:00
ae5b4a940a Rework test mode to use normal LoRa batching and ACK flow 2026-02-20 20:42:06 +01:00
9495e7e8de chore: unify HA manufacturer and add refactor guards 2026-02-18 02:25:07 +01:00
00b2eb859a refactor: move receiver role logic into receiver_pipeline 2026-02-18 02:20:33 +01:00
56960e05e2 refactor: move sender role logic into sender_state_machine 2026-02-18 02:17:12 +01:00
92ac7e8810 receiver: store CSVs by local date and keep UTC history fallback 2026-02-18 01:34:47 +01:00
6f359b11d3 sender: drain backlog with ack-gated catch-up sends 2026-02-18 01:26:43 +01:00
3aff6ea666 Reset sender fault stats on first sync and hourly boundary 2026-02-17 01:27:04 +01:00
e0f3ffc21c Throttle hot-path debug logging for sender stability 2026-02-17 01:15:35 +01:00
d7b5bb0f0b Adapt sender ACK receive window based on observed timing 2026-02-17 01:14:38 +01:00
07d0e6c3e0 Cache encoded inflight payload for retry efficiency 2026-02-17 01:13:27 +01:00
cc5881974c Add sender-local serial diagnostics for pipeline health 2026-02-17 01:12:31 +01:00
ea3e99f350 Use fixed-point meter parsing and early-exit on complete frame 2026-02-17 01:10:43 +01:00
557420c200 Refactor meter parser to single-pass OBIS dispatch 2026-02-17 01:09:56 +01:00
16b8827dca Use local hh:mm:ss in SD CSV human-readable column 2026-02-17 00:39:23 +01:00
318b81adbe Update HA discovery IDs and manufacturer metadata 2026-02-17 00:37:38 +01:00
3ef13bf865 Separate plausible clock from explicit sync state 2026-02-17 00:33:45 +01:00
8d42631045 Harden history parsing and single-point chart rendering 2026-02-17 00:32:48 +01:00
5c71bf841a Handle WiFi config save failures safely 2026-02-17 00:31:55 +01:00
96192d4b2f Fix HA discovery device identifiers format 2026-02-17 00:30:36 +01:00
16aad906a0 Harden receiver ingest against unknown senders 2026-02-17 00:30:06 +01:00
6ea8d9d5fc Use configured local timezone in web UI and drop legacy history CSV parsing 2026-02-16 11:14:30 +01:00
4de1dda82b Add human-readable UTC time alongside epoch in web UI and CSV 2026-02-16 08:57:16 +01:00
5e27e2e7e8 Add meter Sekundenindex anchoring for epoch timestamps
Parse 0-0:96.8.0*255 meter seconds, derive sample epoch from anchored offset, and detect meter-time jumps via monotonic/delta checks.
2026-02-13 23:57:34 +01:00
c58e1627f4 Switch LoRa batch payload to present-mask schema v3
BREAKING CHANGE: schema v2 is no longer supported.

Replaces fixed dt_s timing with a 30-bit present_mask while keeping MQTT JSON unchanged.
2026-02-13 23:57:34 +01:00
7ad0a16a8d Improve meter ingestion resilience under UART gaps 2026-02-13 23:57:34 +01:00
78a880e56f Track receiver duplicate batches in web and OLED 2026-02-13 23:57:34 +01:00
1981a91415 Remove legacy state JSON size cutoff and raise doc capacity 2026-02-13 23:57:34 +01:00
9f5ad5f47e Backfill missed sender sample intervals to keep dt_s consistent 2026-02-13 23:57:34 +01:00
f65a6d28d9 Remove auto-reboot and make timezone configurable 2026-02-13 23:57:34 +01:00
32851ea61b Decouple sender meter reads and harden meter RX robustness 2026-02-13 23:57:34 +01:00
e569c8d627 Add web last-update timestamp and debug auto-reboot 2026-02-13 23:57:34 +01:00
3951183954 Fix sender stale sample reuse and add append helper 2026-02-13 23:57:34 +01:00
db18c549ea Reset RX signal state at start of each receive window 2026-02-13 23:57:34 +01:00
bd3f89a374 Drain oversized LoRa packets to prevent RX FIFO corruption 2026-02-13 23:57:34 +01:00
194c8a40cc Add detailed sender ACK RX diagnostics with reject context 2026-02-13 23:57:34 +01:00
780cf8dc97 Use protocol constants for ACK airtime window sizing 2026-02-13 23:57:34 +01:00
a279c219ae Refactor LoRa protocol to batch+ack with ACK-based time bootstrap 2026-02-13 23:57:34 +01:00
5ad5d3a0cc Normalize power/energy output formatting 2026-02-13 23:57:34 +01:00
43c572a111 Scale ACK RX window to LoRa airtime
- Compute ACK receive window from airtime with bounds and margin
- Retry once if initial window misses
- Document ACK window sizing
2026-02-13 23:57:34 +01:00
06847a9da4 Add RX reject reasons to telemetry and UI
BACKWARD-INCOMPATIBLE: MeterBatch schema bumped to v2 with err_rx_reject.
- Track and log RX reject reasons (CRC/protocol/role/payload/length/id/batch)
- Include rx_reject in sender telemetry JSON and receiver web UI
- Add lora_receive reject reason logging under SERIAL_DEBUG_MODE
2026-02-13 23:57:34 +01:00