Commit Graph
100 Commits
Author SHA1 Message Date
acidburns 99aae76404 test(meter): add fault-count regression test for meter diagnosis 2026-03-16 16:32:21 +01:00
acidburns 3e9259735e Verify republish scripts compatibility with current CSV/MQTT formats
- Fix documentation: CSV header typo (ts_hms_utc  ts_hms_local)
- Add comprehensive compatibility test suite (test_republish_compatibility.py)
- Both republish_mqtt.py and republish_mqtt_gui.py verified working
- Tests: CSV parsing, MQTT JSON format, legacy compatibility, InfluxDB schema
- All 5/5 compatibility tests passing
- Create detailed compatibility reports and validation documentation
2026-03-11 20:43:09 +01:00
acidburns e89aee7048 Add WiFi reconnection retry logic to recover from unreliable WiFi
- Implement periodic WiFi reconnection attempts when stuck in AP mode
- Add WIFI_RECONNECT_INTERVAL_MS config (default 60s) for configurable retry frequency
- Prevent data loss by automatically attempting to switch back to STA mode
- Maintain AP mode for manual configuration if reconnection fails
- Track WiFi config and last reconnection attempt time in shared state
- Add wifi_try_reconnect_sta() and wifi_restore_ap_mode() helper functions
- Log reconnection attempts and results for debugging
2026-03-11 20:32:15 +01:00
acidburns 7fbaf77806 Merge remote-tracking branch 'origin/lora-refactor' into lora-refactor
Resolve merge conflict in src/main.cpp by accepting the refactored version
from origin. The remote branch includes significant architectural improvements:

- New app context and state machine structures
- Refactored receiver and sender logic
- Library reorganization (dd3_legacy_core, dd3_transport_logic)
- Test framework enhancements
- Code quality improvements

The local WiFi reconnection feature (commit 32cd065) will be
re-integrated if needed in the new architecture.
2026-03-11 17:05:39 +01:00
acidburns 32cd0652c9 Improve reliability and add data recovery tools
## Bug Fixes
- Fix integer overflow potential in history bin allocation (web_server.cpp)
  Using uint64_t for intermediate multiplication prevents overflow with different constants

- Prevent data loss during WiFi failures (main.cpp)
  Device now automatically attempts WiFi reconnection every 30 seconds when in AP mode
  Exits AP mode and resumes MQTT transmission as soon as WiFi becomes available
  Data collection and SD logging continue regardless of connectivity

## New Features
- Add standalone MQTT data republisher for lost data recovery
  - Command-line tool (republish_mqtt.py) with interactive and scripting modes
  - GUI tool (republish_mqtt_gui.py) for user-friendly recovery
  - Rate-limited publishing (5 msg/sec default, configurable 1-100)
  - Manual time range selection or auto-detect missing data via InfluxDB
  - Cross-platform support (Windows, macOS, Linux)
  - Converts SD card CSV exports back to MQTT format

## Documentation
- Add comprehensive code review (CODE_REVIEW.md)
  - 16 detailed security and quality assessments
  - Identifies critical HTTPS/auth gaps, medium-priority overflow issues
  - Confirms absence of buffer overflows and unsafe string functions
  - Grade: B+ with areas for improvement

- Add republisher documentation (REPUBLISH_README.md, REPUBLISH_GUI_README.md)
  - Installation and usage instructions
  - Example commands and scenarios
  - Troubleshooting guide
  - Performance characteristics

## Dependencies
- Add requirements_republish.txt
  - paho-mqtt>=1.6.1
  - influxdb-client>=1.18.0

## Impact
- Eliminates data loss scenario where unreliable WiFi leaves device stuck in AP mode
- Provides recovery mechanism for any historical data missed during outages
- Improves code safety with explicit overflow-resistant arithmetic
- Increases operational visibility with comprehensive code review
2026-03-11 17:01:22 +01:00
acidburns a3c61f9b92 docs: align rust-port requirements with current helper ownership
- Update lora transport helper ownership to reference lora_frame_logic (lora_build_frame, lora_parse_frame, lora_crc16_ccitt).
- Remove sender fault-helper entries (ge_seconds, counters_changed, publish_faults_if_needed) that are implemented receiver-side.
2026-02-20 23:56:33 +01:00
acidburns 0577464ec5 refactor: stabilize legacy-core linking and header ownership
- Make include/ the canonical declarations for data_model/html_util/json_codec and convert dd3_legacy_core header copies to thin forwarders.
- Add stable public forwarders for app_context/receiver_pipeline/sender_state_machine and update refactor smoke test to stop using ../../src includes.
- Force-link dd3_legacy_core from setup() to ensure deterministic PlatformIO LDF linking across firmware envs.
- Refresh docs (README, Requirements, docs/TESTS.md) to reflect current module paths and smoke-test include strategy.
2026-02-20 23:29:50 +01:00
acidburns 25709abf8d docs: add legacy unity test guide 2026-02-20 21:34:52 +01:00
acidburns b8e0733a89 test: add json stability and discovery payload coverage 2026-02-20 21:32:35 +01:00
acidburns ca2cd1880a test: add lora frame and chunk reassembly logic suite 2026-02-20 21:26:51 +01:00
acidburns cef1d184ed test: add payload codec regression suite 2026-02-20 21:22:10 +01:00
acidburns 6acb588069 refactor: move html_util into legacy core library 2026-02-20 21:17:48 +01:00
acidburns 2cfdc719c2 test: expand legacy html_util coverage 2026-02-20 21:13:48 +01:00
acidburns ae5b4a940a Rework test mode to use normal LoRa batching and ACK flow 2026-02-20 20:42:06 +01:00
acidburns 1169eab626 Consolidate Rust port requirements and remove refactor notes docs 2026-02-20 20:13:25 +01:00
acidburns 9495e7e8de chore: unify HA manufacturer and add refactor guards 2026-02-18 02:25:07 +01:00
acidburns 00b2eb859a refactor: move receiver role logic into receiver_pipeline 2026-02-18 02:20:33 +01:00
acidburns 56960e05e2 refactor: move sender role logic into sender_state_machine 2026-02-18 02:17:12 +01:00
acidburns 9d7f2ae076 refactor-main-split-baseline 2026-02-18 02:08:27 +01:00
acidburns 53cc982566 docs: reflect sender catch-up and local-date CSV/history behavior 2026-02-18 01:36:20 +01:00
acidburns 92ac7e8810 receiver: store CSVs by local date and keep UTC history fallback 2026-02-18 01:34:47 +01:00
acidburns 6f359b11d3 sender: drain backlog with ack-gated catch-up sends 2026-02-18 01:26:43 +01:00
acidburns 1bdae03cc4 Document hourly and first-sync sender fault counter resets 2026-02-17 01:28:20 +01:00
acidburns 3aff6ea666 Reset sender fault stats on first sync and hourly boundary 2026-02-17 01:27:04 +01:00
acidburns d327f9b68a Document sender efficiency and reliability improvements 2026-02-17 01:17:35 +01:00
acidburns e0f3ffc21c Throttle hot-path debug logging for sender stability 2026-02-17 01:15:35 +01:00
acidburns d7b5bb0f0b Adapt sender ACK receive window based on observed timing 2026-02-17 01:14:38 +01:00
acidburns 07d0e6c3e0 Cache encoded inflight payload for retry efficiency 2026-02-17 01:13:27 +01:00
acidburns cc5881974c Add sender-local serial diagnostics for pipeline health 2026-02-17 01:12:31 +01:00
acidburns ea3e99f350 Use fixed-point meter parsing and early-exit on complete frame 2026-02-17 01:10:43 +01:00
acidburns 557420c200 Refactor meter parser to single-pass OBIS dispatch 2026-02-17 01:09:56 +01:00
acidburns c33fb3274c Update docs for auth, time, discovery, and history changes 2026-02-17 00:45:39 +01:00
acidburns fea3749a93 Require AP web authentication by default 2026-02-17 00:39:44 +01:00
acidburns 16b8827dca Use local hh:mm:ss in SD CSV human-readable column 2026-02-17 00:39:23 +01:00
acidburns 318b81adbe Update HA discovery IDs and manufacturer metadata 2026-02-17 00:37:38 +01:00
acidburns 3ef13bf865 Separate plausible clock from explicit sync state 2026-02-17 00:33:45 +01:00
acidburns 8d42631045 Harden history parsing and single-point chart rendering 2026-02-17 00:32:48 +01:00
acidburns 5c71bf841a Handle WiFi config save failures safely 2026-02-17 00:31:55 +01:00
acidburns 96192d4b2f Fix HA discovery device identifiers format 2026-02-17 00:30:36 +01:00
acidburns 16aad906a0 Harden receiver ingest against unknown senders 2026-02-17 00:30:06 +01:00
acidburns ee849433c8 Refresh README and add firmware requirements for Rust port 2026-02-16 14:38:50 +01:00
acidburns 6ea8d9d5fc Use configured local timezone in web UI and drop legacy history CSV parsing 2026-02-16 11:14:30 +01:00
acidburns 4de1dda82b Add human-readable UTC time alongside epoch in web UI and CSV 2026-02-16 08:57:16 +01:00
acidburns 0a2e4e5a68 docs: update README to current lora-refactor state 2026-02-13 23:57:34 +01:00
acidburns 5e27e2e7e8 Add meter Sekundenindex anchoring for epoch timestamps
Parse 0-0:96.8.0*255 meter seconds, derive sample epoch from anchored offset, and detect meter-time jumps via monotonic/delta checks.
2026-02-13 23:57:34 +01:00
acidburns c58e1627f4 Switch LoRa batch payload to present-mask schema v3
BREAKING CHANGE: schema v2 is no longer supported.

Replaces fixed dt_s timing with a 30-bit present_mask while keeping MQTT JSON unchanged.
2026-02-13 23:57:34 +01:00
acidburns 7ad0a16a8d Improve meter ingestion resilience under UART gaps 2026-02-13 23:57:34 +01:00
acidburns 78a880e56f Track receiver duplicate batches in web and OLED 2026-02-13 23:57:34 +01:00
acidburns 1981a91415 Remove legacy state JSON size cutoff and raise doc capacity 2026-02-13 23:57:34 +01:00
acidburns 9f5ad5f47e Backfill missed sender sample intervals to keep dt_s consistent 2026-02-13 23:57:34 +01:00
acidburns f65a6d28d9 Remove auto-reboot and make timezone configurable 2026-02-13 23:57:34 +01:00
acidburns 32851ea61b Decouple sender meter reads and harden meter RX robustness 2026-02-13 23:57:34 +01:00
acidburns e569c8d627 Add web last-update timestamp and debug auto-reboot 2026-02-13 23:57:34 +01:00
acidburns 3951183954 Fix sender stale sample reuse and add append helper 2026-02-13 23:57:34 +01:00
acidburns db18c549ea Reset RX signal state at start of each receive window 2026-02-13 23:57:34 +01:00
acidburns bd3f89a374 Drain oversized LoRa packets to prevent RX FIFO corruption 2026-02-13 23:57:34 +01:00
acidburns 194c8a40cc Add detailed sender ACK RX diagnostics with reject context 2026-02-13 23:57:34 +01:00
acidburns 780cf8dc97 Use protocol constants for ACK airtime window sizing 2026-02-13 23:57:34 +01:00
acidburns b056b2035a Document minimal batch/ack protocol and timestamp safety rules 2026-02-13 23:57:34 +01:00
acidburns a279c219ae Refactor LoRa protocol to batch+ack with ACK-based time bootstrap 2026-02-13 23:57:34 +01:00
acidburns 5ad5d3a0cc Normalize power/energy output formatting 2026-02-13 23:57:34 +01:00
acidburns 43c572a111 Scale ACK RX window to LoRa airtime
- Compute ACK receive window from airtime with bounds and margin
- Retry once if initial window misses
- Document ACK window sizing
2026-02-13 23:57:34 +01:00
acidburns 06847a9da4 Add RX reject reasons to telemetry and UI
BACKWARD-INCOMPATIBLE: MeterBatch schema bumped to v2 with err_rx_reject.
- Track and log RX reject reasons (CRC/protocol/role/payload/length/id/batch)
- Include rx_reject in sender telemetry JSON and receiver web UI
- Add lora_receive reject reason logging under SERIAL_DEBUG_MODE
2026-02-13 23:57:34 +01:00
acidburns 8681f37fc1 Repeat batch ACKs to cover RX latency
- Add ACK_REPEAT_COUNT/ACK_REPEAT_DELAY_MS and repeat ACK sends
- Update README with repeat-ACK behavior
2026-02-13 23:57:34 +01:00
acidburns 2aca446860 Add LoRa TX timing diagnostics
- Log idle/begin/write/end timing for LoRa TX under SERIAL_DEBUG_MODE
- Document TX timing logs in README
2026-02-13 23:57:34 +01:00
acidburns 8aa503c450 Send batch ACKs immediately after reassembly
- Move ACK ahead of MQTT/web work to meet sender 400ms window
- Update ACK log format and document early-ACK behavior
2026-02-13 23:57:34 +01:00
acidburns b2d7d77f5c Log ACK transmit and reject cases
- Add debug log for ACK TX with batch/sender/receiver ids
- Log rejected ACKs to help diagnose mismatched ids or batches
2026-02-13 23:57:34 +01:00
acidburns c2e1268f1f Improve timesync acquisition and logging
- Add boot acquisition mode with wider RX windows until first TimeSync
- Log sender TimeSync RX results and receiver TX events
- Document acquisition behavior
2026-02-13 23:57:34 +01:00
acidburns ecb73679b6 Validate RTC epoch before setting time
- Reject out-of-range DS3231 epochs and log accept/reject under SERIAL_DEBUG_MODE
- Document RTC validation so LoRa TimeSync can recover
2026-02-13 23:57:34 +01:00
acidburns bd784e88ba Gate slow timesync on LoRa reception
- Keep sender in fast TimeSync listen mode until it receives a LoRa beacon
- Reset scheduler when interval changes to avoid stuck timing
2026-02-13 23:57:34 +01:00
acidburns 20348c3e6b Expose timesync error in MQTT and web UI
BACKWARD-INCOMPATIBLE: MQTT faults payload now always includes err_last/err_last_text and err_last_age (schema change).
2026-02-13 23:57:34 +01:00
acidburns deb060fd20 Add timesync burst handling and sender-only timeout
- Add TimeSync fault code and labels in UI/SD/web docs
- Trigger receiver beacon bursts on sender drift, but keep errors sender-local
- Sender flags TimeSync only after TIME_SYNC_ERROR_TIMEOUT_MS
2026-02-13 23:57:34 +01:00
acidburns e7318f2e71 Keep receiver timesync fast and extend sender fast window
- Receiver now sends time sync every 60s indefinitely (mains powered)
- Sender stays in fast timesync listen mode for first 60s even with RTC
2026-02-13 23:57:34 +01:00
acidburns 532a9154b1 Calibrate battery ADC and document LiPo curve
- Add BATTERY_CAL config and debug logging for raw ADC samples
- Use LiPo voltage curve (4.2V full, 2.9V empty) for % mapping
- Document battery calibration, curve, and debug output in README
2026-02-13 23:57:18 +01:00
acidburns 096b4384d0 Average battery ADC samples
- Read battery 5 times and average for a steadier voltage estimate
2026-02-13 23:57:18 +01:00
acidburns e045673a3a Fix OLED autosleep timing and battery sampling cadence
- Track last OLED activity to avoid double timeout; keep power gating on transitions
- Copy TZ before setenv() in timegm_fallback to avoid invalid pointer reuse
- Add BATTERY_SAMPLE_INTERVAL_MS and only refresh cache at batch start when due
- Keep battery sampling to a single ADC read (Arduino core lacks explicit ADC power gating)
2026-02-13 23:57:18 +01:00
acidburns 532e51a76b Keep receiver LoRa in continuous RX
- Add lora_receive_continuous() helper and use it after init and TX (ACK/time sync)

- Ensure receiver returns to RX immediately after lora_send

- Document continuous RX behavior in README
2026-02-13 23:56:36 +01:00
acidburns 4b5c4e245e Make IEC 62056-21 meter input non-blocking
- Add RX state machine with frame buffer, timeouts, and debug counters

- Expose meter_poll_frame/meter_parse_frame and reuse existing OBIS parsing

- Use cached last-valid frame at 1 Hz sampling to avoid blocking

- Document non-blocking meter handling in README
2026-02-13 23:56:36 +01:00
acidburns 9c5b8fcdb4 Reduce sender power draw (RX windows + CPU/WiFi/ADC/pins)
- Add LoRa idle/sleep/receive-window helpers and use short RX windows for ACK/time sync

- Schedule sender time-sync windows (fast/slow) and track RX vs sleep time in debug

- Lower sender power (80 MHz CPU, WiFi/BT off, reduced ADC sampling, unused pins pulldown)

- Make SERIAL_DEBUG_MODE a build flag, add prod envs with debug off, and document changes
2026-02-13 23:56:36 +01:00
acidburns 4ff5fd1d55 Harden history device ID validation and SD download filename 2026-02-13 23:56:36 +01:00
acidburns bfcb2463c3 Harden web UI auth, input handling, and SD path validation
- Add optional Basic Auth with NVS-backed credentials and STA/AP flags; protect status, wifi, history, and download routes

- Stop pre-filling WiFi/MQTT/Web UI password fields; keep stored secrets on blank and add clear-password checkboxes

- Add HTML escaping + URL encoding helpers and apply to user-controlled strings; add unit test

- Harden /sd/download path validation (prefix, length, dotdot, slashes) and log rejections

- Enforce protocol version in LoRa receive and release GPIO14 before SD init

- Update README security, SD, and GPIO sharing notes
2026-02-13 23:56:36 +01:00
acidburns da16c59690 Add SD history UI and pin remap
- Add SD history chart + download listing to web UI
- Use HSPI for SD and fix SD pin mapping
- Swap role/OLED control pins and update role detection
- Update README pin mapping and SD/history docs
2026-02-13 23:56:36 +01:00
acidburns 6b1ed5b557 Move AP credentials to config and clarify STA UI access 2026-02-13 23:56:36 +01:00
acidburns ff6ade2760 Add SD logging and update docs
- Add optional microSD CSV logging per sender/day on receiver
- Wire logger into receiver packet handling
- Document new batch header fields, build envs, and SD logging
- Make sender links open in a new tab
2026-02-13 23:56:36 +01:00
acidburns bfbe480dad Improve receiver web UI fields and manual 2026-02-13 23:56:36 +01:00
acidburns f2f1949ad0 Include sender error counters in batch payload 2026-02-13 23:56:36 +01:00
acidburns 8ce5f4bc31 Tidy sender page layout and use SF12 2026-02-13 23:56:36 +01:00
acidburns e229efd427 Add payload codec test envs and enable serial debug 2026-02-13 23:56:36 +01:00
acidburns ea68ec699b Update README for binary batch payload and SF11 2026-02-13 23:56:36 +01:00
acidburns aadb520f9d Use compact binary payload for LoRa batches 2026-02-13 23:56:36 +01:00
acidburns e06d431e78 adjust batch ack timing and rename e_wh field 2026-02-13 23:56:36 +01:00
acidburns e84cd999b2 expand web ui with batch table and manual 2026-02-13 23:56:36 +01:00
acidburns 6e3ea1f50a document batching updates and restore bat_v in batches 2026-02-13 23:56:36 +01:00
acidburns cb7527ceeb increase lora throughput and improve receiver display 2026-02-13 23:56:36 +01:00
acidburns cb67febd93 force watchdog reinit for custom timeout 2026-02-13 23:56:36 +01:00
acidburns cd913d4e50 add lora send bypass for debugging 2026-02-13 23:56:36 +01:00
acidburns 7b5cea1c5c instrument tx timings for watchdog analysis 2026-02-13 23:56:36 +01:00
acidburns 8050992817 prevent watchdog from killing while printing json 2026-02-13 23:56:36 +01:00
acidburns 934012cacd serial debugging console implemented, enable via config.h 2026-02-13 23:56:36 +01:00
acidburns 41893cc6ae no sleep while ack pending 2026-02-13 23:56:36 +01:00