Commit Graph

118 Commits

Author SHA1 Message Date
3e9259735e Verify republish scripts compatibility with current CSV/MQTT formats
- Fix documentation: CSV header typo (ts_hms_utc  ts_hms_local)
- Add comprehensive compatibility test suite (test_republish_compatibility.py)
- Both republish_mqtt.py and republish_mqtt_gui.py verified working
- Tests: CSV parsing, MQTT JSON format, legacy compatibility, InfluxDB schema
- All 5/5 compatibility tests passing
- Create detailed compatibility reports and validation documentation
2026-03-11 20:43:09 +01:00
e89aee7048 Add WiFi reconnection retry logic to recover from unreliable WiFi
- Implement periodic WiFi reconnection attempts when stuck in AP mode
- Add WIFI_RECONNECT_INTERVAL_MS config (default 60s) for configurable retry frequency
- Prevent data loss by automatically attempting to switch back to STA mode
- Maintain AP mode for manual configuration if reconnection fails
- Track WiFi config and last reconnection attempt time in shared state
- Add wifi_try_reconnect_sta() and wifi_restore_ap_mode() helper functions
- Log reconnection attempts and results for debugging
2026-03-11 20:32:15 +01:00
7fbaf77806 Merge remote-tracking branch 'origin/lora-refactor' into lora-refactor
Resolve merge conflict in src/main.cpp by accepting the refactored version
from origin. The remote branch includes significant architectural improvements:

- New app context and state machine structures
- Refactored receiver and sender logic
- Library reorganization (dd3_legacy_core, dd3_transport_logic)
- Test framework enhancements
- Code quality improvements

The local WiFi reconnection feature (commit 32cd065) will be
re-integrated if needed in the new architecture.
2026-03-11 17:05:39 +01:00
32cd0652c9 Improve reliability and add data recovery tools
## Bug Fixes
- Fix integer overflow potential in history bin allocation (web_server.cpp)
  Using uint64_t for intermediate multiplication prevents overflow with different constants

- Prevent data loss during WiFi failures (main.cpp)
  Device now automatically attempts WiFi reconnection every 30 seconds when in AP mode
  Exits AP mode and resumes MQTT transmission as soon as WiFi becomes available
  Data collection and SD logging continue regardless of connectivity

## New Features
- Add standalone MQTT data republisher for lost data recovery
  - Command-line tool (republish_mqtt.py) with interactive and scripting modes
  - GUI tool (republish_mqtt_gui.py) for user-friendly recovery
  - Rate-limited publishing (5 msg/sec default, configurable 1-100)
  - Manual time range selection or auto-detect missing data via InfluxDB
  - Cross-platform support (Windows, macOS, Linux)
  - Converts SD card CSV exports back to MQTT format

## Documentation
- Add comprehensive code review (CODE_REVIEW.md)
  - 16 detailed security and quality assessments
  - Identifies critical HTTPS/auth gaps, medium-priority overflow issues
  - Confirms absence of buffer overflows and unsafe string functions
  - Grade: B+ with areas for improvement

- Add republisher documentation (REPUBLISH_README.md, REPUBLISH_GUI_README.md)
  - Installation and usage instructions
  - Example commands and scenarios
  - Troubleshooting guide
  - Performance characteristics

## Dependencies
- Add requirements_republish.txt
  - paho-mqtt>=1.6.1
  - influxdb-client>=1.18.0

## Impact
- Eliminates data loss scenario where unreliable WiFi leaves device stuck in AP mode
- Provides recovery mechanism for any historical data missed during outages
- Improves code safety with explicit overflow-resistant arithmetic
- Increases operational visibility with comprehensive code review
2026-03-11 17:01:22 +01:00
a3c61f9b92 docs: align rust-port requirements with current helper ownership
- Update lora transport helper ownership to reference lora_frame_logic (lora_build_frame, lora_parse_frame, lora_crc16_ccitt).
- Remove sender fault-helper entries (ge_seconds, counters_changed, publish_faults_if_needed) that are implemented receiver-side.
2026-02-20 23:56:33 +01:00
0577464ec5 refactor: stabilize legacy-core linking and header ownership
- Make include/ the canonical declarations for data_model/html_util/json_codec and convert dd3_legacy_core header copies to thin forwarders.
- Add stable public forwarders for app_context/receiver_pipeline/sender_state_machine and update refactor smoke test to stop using ../../src includes.
- Force-link dd3_legacy_core from setup() to ensure deterministic PlatformIO LDF linking across firmware envs.
- Refresh docs (README, Requirements, docs/TESTS.md) to reflect current module paths and smoke-test include strategy.
2026-02-20 23:29:50 +01:00
25709abf8d docs: add legacy unity test guide 2026-02-20 21:34:52 +01:00
b8e0733a89 test: add json stability and discovery payload coverage 2026-02-20 21:32:35 +01:00
ca2cd1880a test: add lora frame and chunk reassembly logic suite 2026-02-20 21:26:51 +01:00
cef1d184ed test: add payload codec regression suite 2026-02-20 21:22:10 +01:00
6acb588069 refactor: move html_util into legacy core library 2026-02-20 21:17:48 +01:00
2cfdc719c2 test: expand legacy html_util coverage 2026-02-20 21:13:48 +01:00
ae5b4a940a Rework test mode to use normal LoRa batching and ACK flow 2026-02-20 20:42:06 +01:00
1169eab626 Consolidate Rust port requirements and remove refactor notes docs 2026-02-20 20:13:25 +01:00
9495e7e8de chore: unify HA manufacturer and add refactor guards 2026-02-18 02:25:07 +01:00
00b2eb859a refactor: move receiver role logic into receiver_pipeline 2026-02-18 02:20:33 +01:00
56960e05e2 refactor: move sender role logic into sender_state_machine 2026-02-18 02:17:12 +01:00
9d7f2ae076 refactor-main-split-baseline 2026-02-18 02:08:27 +01:00
53cc982566 docs: reflect sender catch-up and local-date CSV/history behavior 2026-02-18 01:36:20 +01:00
92ac7e8810 receiver: store CSVs by local date and keep UTC history fallback 2026-02-18 01:34:47 +01:00
6f359b11d3 sender: drain backlog with ack-gated catch-up sends 2026-02-18 01:26:43 +01:00
1bdae03cc4 Document hourly and first-sync sender fault counter resets 2026-02-17 01:28:20 +01:00
3aff6ea666 Reset sender fault stats on first sync and hourly boundary 2026-02-17 01:27:04 +01:00
d327f9b68a Document sender efficiency and reliability improvements 2026-02-17 01:17:35 +01:00
e0f3ffc21c Throttle hot-path debug logging for sender stability 2026-02-17 01:15:35 +01:00
d7b5bb0f0b Adapt sender ACK receive window based on observed timing 2026-02-17 01:14:38 +01:00
07d0e6c3e0 Cache encoded inflight payload for retry efficiency 2026-02-17 01:13:27 +01:00
cc5881974c Add sender-local serial diagnostics for pipeline health 2026-02-17 01:12:31 +01:00
ea3e99f350 Use fixed-point meter parsing and early-exit on complete frame 2026-02-17 01:10:43 +01:00
557420c200 Refactor meter parser to single-pass OBIS dispatch 2026-02-17 01:09:56 +01:00
c33fb3274c Update docs for auth, time, discovery, and history changes 2026-02-17 00:45:39 +01:00
fea3749a93 Require AP web authentication by default 2026-02-17 00:39:44 +01:00
16b8827dca Use local hh:mm:ss in SD CSV human-readable column 2026-02-17 00:39:23 +01:00
318b81adbe Update HA discovery IDs and manufacturer metadata 2026-02-17 00:37:38 +01:00
3ef13bf865 Separate plausible clock from explicit sync state 2026-02-17 00:33:45 +01:00
8d42631045 Harden history parsing and single-point chart rendering 2026-02-17 00:32:48 +01:00
5c71bf841a Handle WiFi config save failures safely 2026-02-17 00:31:55 +01:00
96192d4b2f Fix HA discovery device identifiers format 2026-02-17 00:30:36 +01:00
16aad906a0 Harden receiver ingest against unknown senders 2026-02-17 00:30:06 +01:00
ee849433c8 Refresh README and add firmware requirements for Rust port 2026-02-16 14:38:50 +01:00
6ea8d9d5fc Use configured local timezone in web UI and drop legacy history CSV parsing 2026-02-16 11:14:30 +01:00
4de1dda82b Add human-readable UTC time alongside epoch in web UI and CSV 2026-02-16 08:57:16 +01:00
0a2e4e5a68 docs: update README to current lora-refactor state 2026-02-13 23:57:34 +01:00
5e27e2e7e8 Add meter Sekundenindex anchoring for epoch timestamps
Parse 0-0:96.8.0*255 meter seconds, derive sample epoch from anchored offset, and detect meter-time jumps via monotonic/delta checks.
2026-02-13 23:57:34 +01:00
c58e1627f4 Switch LoRa batch payload to present-mask schema v3
BREAKING CHANGE: schema v2 is no longer supported.

Replaces fixed dt_s timing with a 30-bit present_mask while keeping MQTT JSON unchanged.
2026-02-13 23:57:34 +01:00
7ad0a16a8d Improve meter ingestion resilience under UART gaps 2026-02-13 23:57:34 +01:00
78a880e56f Track receiver duplicate batches in web and OLED 2026-02-13 23:57:34 +01:00
1981a91415 Remove legacy state JSON size cutoff and raise doc capacity 2026-02-13 23:57:34 +01:00
9f5ad5f47e Backfill missed sender sample intervals to keep dt_s consistent 2026-02-13 23:57:34 +01:00
f65a6d28d9 Remove auto-reboot and make timezone configurable 2026-02-13 23:57:34 +01:00