acidburns 32cd0652c9 Improve reliability and add data recovery tools
## Bug Fixes
- Fix integer overflow potential in history bin allocation (web_server.cpp)
  Using uint64_t for intermediate multiplication prevents overflow with different constants

- Prevent data loss during WiFi failures (main.cpp)
  Device now automatically attempts WiFi reconnection every 30 seconds when in AP mode
  Exits AP mode and resumes MQTT transmission as soon as WiFi becomes available
  Data collection and SD logging continue regardless of connectivity

## New Features
- Add standalone MQTT data republisher for lost data recovery
  - Command-line tool (republish_mqtt.py) with interactive and scripting modes
  - GUI tool (republish_mqtt_gui.py) for user-friendly recovery
  - Rate-limited publishing (5 msg/sec default, configurable 1-100)
  - Manual time range selection or auto-detect missing data via InfluxDB
  - Cross-platform support (Windows, macOS, Linux)
  - Converts SD card CSV exports back to MQTT format

## Documentation
- Add comprehensive code review (CODE_REVIEW.md)
  - 16 detailed security and quality assessments
  - Identifies critical HTTPS/auth gaps, medium-priority overflow issues
  - Confirms absence of buffer overflows and unsafe string functions
  - Grade: B+ with areas for improvement

- Add republisher documentation (REPUBLISH_README.md, REPUBLISH_GUI_README.md)
  - Installation and usage instructions
  - Example commands and scenarios
  - Troubleshooting guide
  - Performance characteristics

## Dependencies
- Add requirements_republish.txt
  - paho-mqtt>=1.6.1
  - influxdb-client>=1.18.0

## Impact
- Eliminates data loss scenario where unreliable WiFi leaves device stuck in AP mode
- Provides recovery mechanism for any historical data missed during outages
- Improves code safety with explicit overflow-resistant arithmetic
- Increases operational visibility with comprehensive code review
2026-03-11 17:01:22 +01:00
2026-01-20 01:39:06 +01:00
2026-01-23 17:10:04 +01:00
2026-01-20 01:39:06 +01:00
2026-01-20 01:39:06 +01:00

DD3-LoRa-Bridge-MultiSender

Firmware for LilyGO T3 v1.6.1 (ESP32 + SX1276 + SSD1306) that runs in two roles:

  • Sender (GPIO14 HIGH): reads one IEC 62056-21 meter, builds 30-slot sparse batches, sends via LoRa.
  • Receiver (GPIO14 LOW): receives/ACKs batches, publishes MQTT, serves web UI, logs to SD.

Architecture Summary

  • Single codebase, role selected at boot by detect_role() (src/config.cpp).
  • LoRa transport is wrapped with firmware-level CRC16-CCITT (src/lora_transport.cpp).
  • Sender meter ingest is decoupled from LoRa waits via FreeRTOS meter reader task + queue on ESP32 (src/main.cpp).
  • Batch payload codec is schema v3 with a 30-bit present_mask over [t_last-29, t_last] (src/payload_codec.cpp).
  • Sender only starts normal metering/transmit flow after valid time bootstrap from receiver ACK.
  • Receiver runs STA mode if stored config is valid and connects, otherwise AP fallback.

LoRa Protocol

On-air frame:

[msg_kind:1][device_short_id:2][payload...][crc16:2]

msg_kind:

  • 0: BatchUp
  • 1: AckDown

BatchUp

Transport layer chunks payload into:

[batch_id_le:2][chunk_index:1][chunk_count:1][total_len_le:2][chunk_payload...]

Receiver reassembles all chunks before decode.

Payload codec (schema=3, magic 0xDDB3) carries:

  • metadata: sender ID, batch ID, t_last, present_mask, battery mV, error counters
  • arrays per present sample: energy_wh[], p1_w[], p2_w[], p3_w[]

n == 0 with present_mask == 0 is valid and used for sync request packets.

AckDown (7 bytes payload)

[flags:1][batch_id_be:2][epoch_utc_be:4]

  • flags bit0: time_valid
  • ACK is repeated (ACK_REPEAT_COUNT=3, ACK_REPEAT_DELAY_MS=200)
  • Sender sets local time only if time_valid=1 and epoch >= MIN_ACCEPTED_EPOCH_UTC (2026-02-01 00:00:00 UTC)

Time Bootstrap and Timezone

Sender boot starts in sync-only mode:

  • g_time_acquired=false
  • sends sync requests every SYNC_REQUEST_INTERVAL_MS (15s)
  • does not run normal 1 Hz sample/batch flow yet

After valid ACK time:

  • time_set_utc() is called
  • g_time_acquired=true
  • normal 1 Hz sampling + periodic batch transmission starts

Timezone:

  • TIMEZONE_TZ from include/config.h is applied in time_manager.
  • Web/OLED local-time rendering uses this timezone.
  • Default: CET-1CEST,M3.5.0/2,M10.5.0/3.

Sender Meter Path

Implemented by src/meter_driver.cpp and sender loop in src/main.cpp:

  • UART: Serial2, GPIO34, 9600 7E1
  • ESP32 RX buffer enlarged to 8192
  • Frame detection / ... !, timeout METER_FRAME_TIMEOUT_MS=3000
  • Parsed OBIS fields:
    • 0-0:96.8.0*255 meter Sekundenindex (hex u32)
    • 1-0:1.8.0 total energy (auto scales Wh -> kWh when unit is Wh)
    • 1-0:16.7.0 total active power
    • 1-0:36.7.0, 56.7.0, 76.7.0 phase powers

Timestamp derivation:

  • anchor offset: epoch_offset = epoch_now - meter_seconds
  • sample epoch: ts_utc = meter_seconds + epoch_offset
  • jump checks: rollback, wall-time delta mismatch, anchor drift

Sender builds sparse 30-slot windows and sends every METER_SEND_INTERVAL_MS (30s).

Receiver Behavior

For decoded BatchUp:

  1. Reassemble and decode.
  2. Send AckDown immediately.
  3. Track duplicates per configured sender (EXPECTED_SENDER_IDS).
  4. If duplicate: update duplicate counters/time, skip data write/publish.
  5. If n==0: sync request path only.
  6. Else reconstruct each sample timestamp from t_last + present_mask, then:
    • append to SD CSV
    • publish MQTT state
    • update web status and last batch table

MQTT

State topic:

  • smartmeter/<device_id>/state

Fault topic (retained):

  • smartmeter/<device_id>/faults

State JSON (src/json_codec.cpp) includes:

  • id, ts, e_kwh
  • p_w, p1_w, p2_w, p3_w
  • bat_v, bat_pct
  • optional link: rssi, snr
  • err_last, rx_reject, rx_reject_text
  • non-zero fault counters when available

Home Assistant discovery:

  • enabled by ENABLE_HA_DISCOVERY=true
  • publishes to homeassistant/sensor/<device_id>/<key>/config

Web UI, Wi-Fi, SD

  • Wi-Fi/MQTT/NTP/web-auth config is stored in Preferences.
  • AP fallback SSID prefix: DD3-Bridge-.
  • Default web credentials: admin/admin.
  • AP auth requirement is controlled by WEB_AUTH_REQUIRE_AP (default false).
  • STA auth requirement is controlled by WEB_AUTH_REQUIRE_STA (default true).

Web timestamp display:

  • human-facing timestamps show epoch (HH:MM:SS TZ) in local configured timezone.

SD CSV logging (src/sd_logger.cpp):

  • header: ts_utc,ts_hms_utc,p_w,p1_w,p2_w,p3_w,e_kwh,bat_v,bat_pct,rssi,snr,err_m,err_d,err_tx,err_last
  • ts_hms_utc is UTC HH:MM:SS

History parser (src/web_server.cpp):

  • expects the current CSV layout above
  • legacy CSV layouts are not parsed (no backward compatibility)

OLED duplicate display:

  • receiver sender-pages show duplicate rate as pct (absolute) and last duplicate as HH:MM.

Build Environments

From platformio.ini:

  • lilygo-t3-v1-6-1
  • lilygo-t3-v1-6-1-test
  • lilygo-t3-v1-6-1-868
  • lilygo-t3-v1-6-1-868-test
  • lilygo-t3-v1-6-1-payload-test
  • lilygo-t3-v1-6-1-868-payload-test
  • lilygo-t3-v1-6-1-prod
  • lilygo-t3-v1-6-1-868-prod

Example:

python -m platformio run -e lilygo-t3-v1-6-1

Test Mode

ENABLE_TEST_MODE replaces normal loops with test_sender_loop / test_receiver_loop (src/test_mode.cpp):

  • Sender emits periodic JSON test payloads over LoRa.
  • Receiver decodes test payloads, updates display test codes, publishes MQTT to:
    • smartmeter/<device_id>/test
Description
Unified Firmware for the LilyGO T3 v1.6.1 433MHz Version. Sender will read DD3 Smart Meter Data and send it to reciever as well as display on OLED. Reciever will publish to mqtt and show on a local website as well as OLED.
Readme 2.8 MiB
Languages
C++ 95.6%
C 4.4%