Commit Graph

80 Commits

Author SHA1 Message Date
32cd0652c9 Improve reliability and add data recovery tools
## Bug Fixes
- Fix integer overflow potential in history bin allocation (web_server.cpp)
  Using uint64_t for intermediate multiplication prevents overflow with different constants

- Prevent data loss during WiFi failures (main.cpp)
  Device now automatically attempts WiFi reconnection every 30 seconds when in AP mode
  Exits AP mode and resumes MQTT transmission as soon as WiFi becomes available
  Data collection and SD logging continue regardless of connectivity

## New Features
- Add standalone MQTT data republisher for lost data recovery
  - Command-line tool (republish_mqtt.py) with interactive and scripting modes
  - GUI tool (republish_mqtt_gui.py) for user-friendly recovery
  - Rate-limited publishing (5 msg/sec default, configurable 1-100)
  - Manual time range selection or auto-detect missing data via InfluxDB
  - Cross-platform support (Windows, macOS, Linux)
  - Converts SD card CSV exports back to MQTT format

## Documentation
- Add comprehensive code review (CODE_REVIEW.md)
  - 16 detailed security and quality assessments
  - Identifies critical HTTPS/auth gaps, medium-priority overflow issues
  - Confirms absence of buffer overflows and unsafe string functions
  - Grade: B+ with areas for improvement

- Add republisher documentation (REPUBLISH_README.md, REPUBLISH_GUI_README.md)
  - Installation and usage instructions
  - Example commands and scenarios
  - Troubleshooting guide
  - Performance characteristics

## Dependencies
- Add requirements_republish.txt
  - paho-mqtt>=1.6.1
  - influxdb-client>=1.18.0

## Impact
- Eliminates data loss scenario where unreliable WiFi leaves device stuck in AP mode
- Provides recovery mechanism for any historical data missed during outages
- Improves code safety with explicit overflow-resistant arithmetic
- Increases operational visibility with comprehensive code review
2026-03-11 17:01:22 +01:00
ee849433c8 Refresh README and add firmware requirements for Rust port 2026-02-16 14:38:50 +01:00
6ea8d9d5fc Use configured local timezone in web UI and drop legacy history CSV parsing 2026-02-16 11:14:30 +01:00
4de1dda82b Add human-readable UTC time alongside epoch in web UI and CSV 2026-02-16 08:57:16 +01:00
0a2e4e5a68 docs: update README to current lora-refactor state 2026-02-13 23:57:34 +01:00
5e27e2e7e8 Add meter Sekundenindex anchoring for epoch timestamps
Parse 0-0:96.8.0*255 meter seconds, derive sample epoch from anchored offset, and detect meter-time jumps via monotonic/delta checks.
2026-02-13 23:57:34 +01:00
c58e1627f4 Switch LoRa batch payload to present-mask schema v3
BREAKING CHANGE: schema v2 is no longer supported.

Replaces fixed dt_s timing with a 30-bit present_mask while keeping MQTT JSON unchanged.
2026-02-13 23:57:34 +01:00
7ad0a16a8d Improve meter ingestion resilience under UART gaps 2026-02-13 23:57:34 +01:00
78a880e56f Track receiver duplicate batches in web and OLED 2026-02-13 23:57:34 +01:00
1981a91415 Remove legacy state JSON size cutoff and raise doc capacity 2026-02-13 23:57:34 +01:00
9f5ad5f47e Backfill missed sender sample intervals to keep dt_s consistent 2026-02-13 23:57:34 +01:00
f65a6d28d9 Remove auto-reboot and make timezone configurable 2026-02-13 23:57:34 +01:00
32851ea61b Decouple sender meter reads and harden meter RX robustness 2026-02-13 23:57:34 +01:00
e569c8d627 Add web last-update timestamp and debug auto-reboot 2026-02-13 23:57:34 +01:00
3951183954 Fix sender stale sample reuse and add append helper 2026-02-13 23:57:34 +01:00
acidburns
1769949dc8 docs: refresh README for lora-refactor behavior 2026-02-13 23:57:34 +01:00
db18c549ea Reset RX signal state at start of each receive window 2026-02-13 23:57:34 +01:00
bd3f89a374 Drain oversized LoRa packets to prevent RX FIFO corruption 2026-02-13 23:57:34 +01:00
194c8a40cc Add detailed sender ACK RX diagnostics with reject context 2026-02-13 23:57:34 +01:00
780cf8dc97 Use protocol constants for ACK airtime window sizing 2026-02-13 23:57:34 +01:00
b056b2035a Document minimal batch/ack protocol and timestamp safety rules 2026-02-13 23:57:34 +01:00
a279c219ae Refactor LoRa protocol to batch+ack with ACK-based time bootstrap 2026-02-13 23:57:34 +01:00
5ad5d3a0cc Normalize power/energy output formatting 2026-02-13 23:57:34 +01:00
43c572a111 Scale ACK RX window to LoRa airtime
- Compute ACK receive window from airtime with bounds and margin
- Retry once if initial window misses
- Document ACK window sizing
2026-02-13 23:57:34 +01:00
06847a9da4 Add RX reject reasons to telemetry and UI
BACKWARD-INCOMPATIBLE: MeterBatch schema bumped to v2 with err_rx_reject.
- Track and log RX reject reasons (CRC/protocol/role/payload/length/id/batch)
- Include rx_reject in sender telemetry JSON and receiver web UI
- Add lora_receive reject reason logging under SERIAL_DEBUG_MODE
2026-02-13 23:57:34 +01:00
8681f37fc1 Repeat batch ACKs to cover RX latency
- Add ACK_REPEAT_COUNT/ACK_REPEAT_DELAY_MS and repeat ACK sends
- Update README with repeat-ACK behavior
2026-02-13 23:57:34 +01:00
2aca446860 Add LoRa TX timing diagnostics
- Log idle/begin/write/end timing for LoRa TX under SERIAL_DEBUG_MODE
- Document TX timing logs in README
2026-02-13 23:57:34 +01:00
8aa503c450 Send batch ACKs immediately after reassembly
- Move ACK ahead of MQTT/web work to meet sender 400ms window
- Update ACK log format and document early-ACK behavior
2026-02-13 23:57:34 +01:00
b2d7d77f5c Log ACK transmit and reject cases
- Add debug log for ACK TX with batch/sender/receiver ids
- Log rejected ACKs to help diagnose mismatched ids or batches
2026-02-13 23:57:34 +01:00
c2e1268f1f Improve timesync acquisition and logging
- Add boot acquisition mode with wider RX windows until first TimeSync
- Log sender TimeSync RX results and receiver TX events
- Document acquisition behavior
2026-02-13 23:57:34 +01:00
ecb73679b6 Validate RTC epoch before setting time
- Reject out-of-range DS3231 epochs and log accept/reject under SERIAL_DEBUG_MODE
- Document RTC validation so LoRa TimeSync can recover
2026-02-13 23:57:34 +01:00
bd784e88ba Gate slow timesync on LoRa reception
- Keep sender in fast TimeSync listen mode until it receives a LoRa beacon
- Reset scheduler when interval changes to avoid stuck timing
2026-02-13 23:57:34 +01:00
20348c3e6b Expose timesync error in MQTT and web UI
BACKWARD-INCOMPATIBLE: MQTT faults payload now always includes err_last/err_last_text and err_last_age (schema change).
2026-02-13 23:57:34 +01:00
deb060fd20 Add timesync burst handling and sender-only timeout
- Add TimeSync fault code and labels in UI/SD/web docs
- Trigger receiver beacon bursts on sender drift, but keep errors sender-local
- Sender flags TimeSync only after TIME_SYNC_ERROR_TIMEOUT_MS
2026-02-13 23:57:34 +01:00
e7318f2e71 Keep receiver timesync fast and extend sender fast window
- Receiver now sends time sync every 60s indefinitely (mains powered)
- Sender stays in fast timesync listen mode for first 60s even with RTC
2026-02-13 23:57:34 +01:00
532a9154b1 Calibrate battery ADC and document LiPo curve
- Add BATTERY_CAL config and debug logging for raw ADC samples
- Use LiPo voltage curve (4.2V full, 2.9V empty) for % mapping
- Document battery calibration, curve, and debug output in README
2026-02-13 23:57:18 +01:00
096b4384d0 Average battery ADC samples
- Read battery 5 times and average for a steadier voltage estimate
2026-02-13 23:57:18 +01:00
e045673a3a Fix OLED autosleep timing and battery sampling cadence
- Track last OLED activity to avoid double timeout; keep power gating on transitions
- Copy TZ before setenv() in timegm_fallback to avoid invalid pointer reuse
- Add BATTERY_SAMPLE_INTERVAL_MS and only refresh cache at batch start when due
- Keep battery sampling to a single ADC read (Arduino core lacks explicit ADC power gating)
2026-02-13 23:57:18 +01:00
532e51a76b Keep receiver LoRa in continuous RX
- Add lora_receive_continuous() helper and use it after init and TX (ACK/time sync)

- Ensure receiver returns to RX immediately after lora_send

- Document continuous RX behavior in README
2026-02-13 23:56:36 +01:00
4b5c4e245e Make IEC 62056-21 meter input non-blocking
- Add RX state machine with frame buffer, timeouts, and debug counters

- Expose meter_poll_frame/meter_parse_frame and reuse existing OBIS parsing

- Use cached last-valid frame at 1 Hz sampling to avoid blocking

- Document non-blocking meter handling in README
2026-02-13 23:56:36 +01:00
9c5b8fcdb4 Reduce sender power draw (RX windows + CPU/WiFi/ADC/pins)
- Add LoRa idle/sleep/receive-window helpers and use short RX windows for ACK/time sync

- Schedule sender time-sync windows (fast/slow) and track RX vs sleep time in debug

- Lower sender power (80 MHz CPU, WiFi/BT off, reduced ADC sampling, unused pins pulldown)

- Make SERIAL_DEBUG_MODE a build flag, add prod envs with debug off, and document changes
2026-02-13 23:56:36 +01:00
4ff5fd1d55 Harden history device ID validation and SD download filename 2026-02-13 23:56:36 +01:00
bfcb2463c3 Harden web UI auth, input handling, and SD path validation
- Add optional Basic Auth with NVS-backed credentials and STA/AP flags; protect status, wifi, history, and download routes

- Stop pre-filling WiFi/MQTT/Web UI password fields; keep stored secrets on blank and add clear-password checkboxes

- Add HTML escaping + URL encoding helpers and apply to user-controlled strings; add unit test

- Harden /sd/download path validation (prefix, length, dotdot, slashes) and log rejections

- Enforce protocol version in LoRa receive and release GPIO14 before SD init

- Update README security, SD, and GPIO sharing notes
2026-02-13 23:56:36 +01:00
da16c59690 Add SD history UI and pin remap
- Add SD history chart + download listing to web UI
- Use HSPI for SD and fix SD pin mapping
- Swap role/OLED control pins and update role detection
- Update README pin mapping and SD/history docs
2026-02-13 23:56:36 +01:00
6b1ed5b557 Move AP credentials to config and clarify STA UI access 2026-02-13 23:56:36 +01:00
ff6ade2760 Add SD logging and update docs
- Add optional microSD CSV logging per sender/day on receiver
- Wire logger into receiver packet handling
- Document new batch header fields, build envs, and SD logging
- Make sender links open in a new tab
2026-02-13 23:56:36 +01:00
bfbe480dad Improve receiver web UI fields and manual 2026-02-13 23:56:36 +01:00
f2f1949ad0 Include sender error counters in batch payload 2026-02-13 23:56:36 +01:00
8ce5f4bc31 Tidy sender page layout and use SF12 2026-02-13 23:56:36 +01:00
e229efd427 Add payload codec test envs and enable serial debug 2026-02-13 23:56:36 +01:00