DD3-LoRa-Bridge-MultiSender/README.md

# DD3-LoRa-Bridge-MultiSender

Firmware for LilyGO T3 v1.6.1 (`ESP32 + SX1276 + SSD1306`) that runs in two roles:
- `Sender` (`GPIO14` HIGH): reads one IEC 62056-21 meter, builds 30-slot sparse batches, sends via LoRa.
- `Receiver` (`GPIO14` LOW): receives/ACKs batches, publishes MQTT, serves web UI, logs to SD.

## Architecture Summary

- Single codebase, role selected at boot by `detect_role()` (`src/config.cpp`).
- LoRa transport is wrapped with firmware-level CRC16-CCITT (`src/lora_transport.cpp`).
- Sender meter ingest is decoupled from LoRa waits via FreeRTOS meter reader task + queue on ESP32 (`src/sender_state_machine.cpp`).
- Batch payload codec is schema `v4` with a 30-bit `present_mask` over `[meter_t_last-29, meter_t_last]` and a separate UTC anchor (`lib/dd3_legacy_core/src/payload_codec.cpp`).
- Sender retries reuse cached encoded payload bytes (no re-encode on retry path).
- Sender ACK receive windows adapt from observed ACK RTT + miss streak.
- Sender catch-up mode drains backlog with immediate extra sends when more than one batch is queued (still ACK-gated, single inflight batch).
- Sender only starts normal metering/transmit flow after valid time bootstrap from receiver ACK.
- Sender fault counters are reset at first valid time sync and again at each UTC hour boundary.
- Receiver runs STA mode if stored config is valid and connects, otherwise AP fallback.
- Current bench defaults in `include/config.h`: 868 MHz LoRa, sender short-ID `0x6540`, receiver short-ID `0x7EB4`.

## LoRa Protocol

On-air frame:

`[msg_kind:1][device_short_id:2][payload...][crc16:2]`

`msg_kind`:
- `0`: `BatchUp`
- `1`: `AckDown`

### BatchUp

Transport layer chunks payload into:

`[batch_id_le:2][chunk_index:1][chunk_count:1][total_len_le:2][chunk_payload...]`

Receiver reassembles all chunks before decode.

Payload codec (`schema=4`, magic `0xDDB3`) carries:
- metadata: sender ID, batch ID, `meter_t_last`, `ts_utc_last`, `present_mask`, battery mV, error counters
- arrays per present sample: `energy_wh[]`, `p1_w[]`, `p2_w[]`, `p3_w[]`

`n == 0` with `present_mask == 0` is valid and used for sync request packets.

Schema `v4` is not wire-compatible with schema `v3`: both sender and receiver must be flashed with matching firmware.

### AckDown (7 bytes payload)

`[flags:1][batch_id_be:2][epoch_utc_be:4]`

- `flags bit0`: `time_valid`
- ACK is repeated (`ACK_REPEAT_COUNT=3`, `ACK_REPEAT_DELAY_MS=200`)
- Sender sets local time only if `time_valid=1` and `epoch >= MIN_ACCEPTED_EPOCH_UTC` (`2026-02-01 00:00:00 UTC`)
- Sender ACK wait windows are adaptive (short first window, expanded second window on miss)

## Time Bootstrap and Timezone

Sender boot starts in sync-only mode:
- `g_time_acquired=false`
- sends sync requests every `SYNC_REQUEST_INTERVAL_MS` (`15s`)
- does not run normal 1 Hz sample/batch flow yet

After valid ACK time:
- `time_set_utc()` is called
- `g_time_acquired=true`
- sender fault counters are reset once (`err_m`, `err_d`, `err_tx`, last-error state)
- normal 1 Hz sampling + periodic batch transmission starts

After initial sync:
- sender fault counters are reset again once per UTC hour when the hour index changes (`HH:00 UTC` boundary)

Timezone:
- `TIMEZONE_TZ` from `include/config.h` is applied in `time_manager`.
- Web/OLED local-time rendering uses this timezone.
- Default: `CET-1CEST,M3.5.0/2,M10.5.0/3`.

## Sender Meter Path

Implemented by `src/meter_driver.cpp` and sender loop in `src/sender_state_machine.cpp`:
- UART: `Serial2`, `GPIO34`, `9600 7E1`
- ESP32 RX buffer enlarged to `8192`
- Frame detection `/ ... !`, timeout `METER_FRAME_TIMEOUT_MS=3000`
- Single-pass OBIS line dispatch (no repeated multi-key scans per line)
- Fixed-point decimal parser (dot/comma decimals), with early-exit once all required OBIS fields are captured
- Parsed OBIS fields:
  - `0-0:96.8.0*255` meter Sekundenindex (hex u32)
  - `1-0:1.8.0` total energy (auto scales Wh -> kWh when unit is Wh)
  - `1-0:16.7.0` total active power
  - `1-0:36.7.0`, `56.7.0`, `76.7.0` phase powers

Timestamp derivation:
- anchor offset: `epoch_offset = epoch_now - meter_seconds`
- sample epoch: `ts_utc = meter_seconds + epoch_offset`
- jump checks: rollback, wall-time delta mismatch, anchor drift

Sender builds sparse 30-slot windows in meter-seconds space and sends every `METER_SEND_INTERVAL_MS` (`30s`).
Samples without a valid meter seconds value are rejected for normal batch transmission.
When backlog is present (`batch_q > 1`), sender transmits additional queued batches immediately after ACK to reduce lag, while keeping stop-and-wait ACK semantics.

Sender diagnostics (serial debug mode):
- periodic structured `diag:` line with:
  - meter parser counters (`ok/parse_fail/overflow/timeout`)
  - meter queue stats (`depth/high-watermark/drops`)
  - ACK stats (`last RTT`, `EWMA RTT`, `miss streak`, timeout/retry totals)
  - sender runtime totals (`rx window ms`, `sleep ms`)
- diagnostics are local-only (serial); LoRa payload schema/fields are unchanged.

## Receiver Behavior

For decoded `BatchUp`:
1. Reassemble and decode.
2. Validate sender identity (`EXPECTED_SENDER_IDS` and payload sender ID mapping).
3. Reject unknown/mismatched senders before ACK and before SD/MQTT/web updates.
4. Send `AckDown` promptly for accepted senders.
5. Track duplicates per configured sender.
6. If duplicate: update duplicate counters/time, skip data write/publish.
7. If `n==0`: sync request path only.
8. Else reconstruct each sample from `meter_t_last + present_mask` and `ts_utc_last + present_mask`, then:
   - append to SD CSV
   - publish MQTT state
   - update web status and last batch table

ACK validation accepts only configured sender IDs, the sender's own short-ID, or configured receiver short-IDs (`EXPECTED_RECEIVER_IDS`) so receiver-originated ACKs are allowed without accepting arbitrary device IDs.

## MQTT

State topic:
- `smartmeter/<device_id>/state`

Fault topic (retained):
- `smartmeter/<device_id>/faults`

State JSON (`lib/dd3_legacy_core/src/json_codec.cpp`) includes:
- `id`, `ts`, `e_kwh`
- `p_w`, `p1_w`, `p2_w`, `p3_w`
- `bat_v`, `bat_pct`
- optional link: `rssi`, `snr`
- `err_last`, `rx_reject`, `rx_reject_text`
- non-zero fault counters when available

Sender fault counter lifecycle:
- counters are cumulative only within the current UTC-hour window after first sync
- counters reset on first valid sender time sync and at each subsequent UTC hour boundary

Home Assistant discovery:
- enabled by `ENABLE_HA_DISCOVERY=true`
- publishes to `homeassistant/sensor/<device_id>/<key>/config`
- `unique_id` format is `<device_id>_<key>` (example: `dd3-F19C_energy`)
- device metadata:
  - `identifiers: ["<device_id>"]`
  - `name: "<device_id>"`
  - `model: "DD3-LoRa-Bridge"`
  - `manufacturer: "AcidBurns"` (from `HA_MANUFACTURER` in `include/config.h`)
  - single source of truth: change manufacturer only in `include/config.h`

## Web UI, Wi-Fi, SD

- Wi-Fi/MQTT/NTP/web-auth config is stored in Preferences.
- AP fallback SSID prefix: `DD3-Bridge-`.
- Default web credentials: `admin/admin`.
- AP auth requirement is controlled by `WEB_AUTH_REQUIRE_AP` (default `true`).
- STA auth requirement is controlled by `WEB_AUTH_REQUIRE_STA` (default `true`).
- If the receiver boots into AP mode with saved Wi-Fi and MQTT config, it periodically retries STA mode and reinitializes NTP, MQTT, and the web server after a successful reconnect.

Web timestamp display:
- human-facing timestamps show `epoch (HH:MM:SS TZ)` in local configured timezone.

SD CSV logging (`src/sd_logger.cpp`):
- header: `ts_utc,ts_hms_local,p_w,p1_w,p2_w,p3_w,e_kwh,bat_v,bat_pct,rssi,snr,err_m,err_d,err_tx,err_last`
- `ts_hms_local` is local `HH:MM:SS` derived from `TIMEZONE_TZ`
- per-day file partition uses local date from `TIMEZONE_TZ`: `/dd3/<device_id>/YYYY-MM-DD.csv`

History parser (`src/web_server.cpp`):
- accepts both:
  - current layout (`ts_utc,ts_hms_local,p_w,...`)
  - legacy layout (`ts_utc,p_w,...`)
- daily file lookup prefers local-date filenames and falls back to legacy UTC-date filenames for backward compatibility
- requires full numeric parse for `ts_utc` and `p_w` (rejects trailing junk)

OLED duplicate display:
- receiver sender-pages show duplicate rate as `pct (absolute)` and last duplicate as `HH:MM`.

## Build Environments

From `platformio.ini`:
- `production`: serial debug off, light sleep on
- `debug`: serial diagnostics on, real meter and real LoRa
- `test`: synthetic meter samples and payload codec self-test, serial debug off

Example:

```bash
python -m platformio run -e production
```

## Test Mode

`ENABLE_TEST_MODE` replaces normal loops with `test_sender_loop` / `test_receiver_loop` (`src/test_mode.cpp`):
- Sender emits periodic JSON test payloads over LoRa.
- Receiver decodes test payloads, updates display test codes, publishes MQTT to:
  - `smartmeter/<device_id>/test`