DD3-LoRa-Bridge-MultiSender/README.md

# DD3 LoRa Bridge (Multi-Sender)

Unified firmware for LilyGO T3 v1.6.1 (ESP32 + SX1276 + SSD1306) that runs as **Sender** or **Receiver** based on a GPIO jumper. Senders read DD3 smart meter values and transmit compact binary batches over LoRa. The receiver validates packets, publishes to MQTT, provides a web UI, and shows per-sender status on the OLED.

## Hardware
Board: **LilyGO T3 LoRa32 v1.6.1** (ESP32 + SX1276 + SSD1306 128x64 + LiPo)
Variants:
- SX1276 **433 MHz** module (default build)
- SX1276 **868 MHz** module (use 868 build environments)

### Pin Mapping
- LoRa (SX1276)
  - SCK: GPIO5
  - MISO: GPIO19
  - MOSI: GPIO27
  - NSS/CS: GPIO18
  - RST: GPIO23
  - DIO0: GPIO26
- OLED (SSD1306)
  - SDA: GPIO21
  - SCL: GPIO22
  - RST: **not used** (SSD1306 init uses `-1` reset pin)
  - I2C address: 0x3C
- microSD (on-board)
  - CS: GPIO13
  - MOSI: GPIO15
  - SCK: GPIO14
  - MISO: GPIO2
- I2C RTC (DS3231)
  - SDA: GPIO21
  - SCL: GPIO22
  - I2C address: 0x68
- Battery ADC: GPIO35 (via on-board divider)
- **Role select**: GPIO14 (INPUT_PULLDOWN, sampled at boot, **shared with SD SCK**)
  - HIGH = Sender
  - LOW/floating = Receiver
- **OLED control**: GPIO13 (INPUT_PULLDOWN, sender only, **shared with SD CS**)
  - HIGH = force OLED on
  - LOW = allow auto-off after timeout
  - Not used on receiver (OLED always on)
- Smart meter UART RX: GPIO34 (input-only, always connected)

### Notes on GPIOs
- GPIO34/35/36/39 are input-only and have **no internal pullups/pulldowns**.
- Strap pins (GPIO0/2/4/5/12/15) can affect boot; avoid for role or control jumpers.
- GPIO14 is shared between role select and SD SCK. **Do not attach the role jumper in Receiver mode if the SD card is connected/used**, and never force GPIO14 high when using SD.
- GPIO13 is shared between OLED control and SD CS. Avoid driving OLED control when SD is active.
- Receiver firmware releases GPIO14 to `INPUT` (no pulldown) after boot before SD SPI init.

## Firmware Roles
### Sender (battery-powered)
- Reads smart meter via optical IR (UART 9600 7E1).
- Extracts OBIS values:
  - Energy total: 1-0:1.8.0*255
  - Total power: 1-0:16.7.0*255
  - Phase power: 36.7 / 56.7 / 76.7
- Reads battery voltage and estimates SoC.
- Builds compact binary batch payload, wraps in LoRa packet, transmits.
- Light sleeps between meter reads; batches are sent every 30s.
- Listens for LoRa time sync packets to set UTC clock.
- Uses DS3231 RTC after boot if no time sync has arrived yet.
- OLED shows status + meter data pages.

**Sender flow (pseudo-code)**:
```cpp
void sender_loop() {
  meter_read_every_second(); // OBIS -> MeterData samples
  read_battery(data);        // VBAT + SoC

  if (time_to_send_batch()) {
    payload = encode_batch(samples, batch_id); // compact binary batch
    lora_send(packet(MeterBatch, payload));
  }

  display_set_last_meter(data);
  display_set_last_read(ok);
  display_set_last_tx(ok);
  display_tick();

  lora_receive_time_sync(); // optional
  light_sleep_until_next_event();
}
```

**Key sender functions**:
```cpp
bool meter_read(MeterData &data);        // parse OBIS fields
void read_battery(MeterData &data);      // ADC -> volts + percent
bool meterDataToJson(const MeterData&, String&);
bool compressBuffer(const uint8_t*, size_t, uint8_t*, size_t, size_t&); // MeterData only
bool lora_send(const LoraPacket &pkt);   // add header + CRC16 and transmit
```

### Receiver (USB-powered)
- WiFi STA connect using stored config; if not available/fails, starts AP.
- NTP sync (UTC) and local display in Europe/Berlin.
- Receives LoRa packets, verifies CRC16, decompresses MeterData JSON, decodes binary batches.
- Publishes meter JSON to MQTT.
- Sends ACKs for MeterBatch packets and de-duplicates by batch_id.
- Web UI:
  - AP mode: status + WiFi/MQTT config.
  - STA mode: status + per-sender pages.
- OLED cycles through receiver status and per-sender pages (receiver OLED never sleeps).

**Receiver loop (pseudo-code)**:
```cpp
void receiver_loop() {
  if (lora_receive(pkt)) {
    if (pkt.type == MeterData) {
      json = decompressBuffer(pkt.payload);
      if (jsonToMeterData(json, data)) {
        update_sender_status(data);
        mqtt_publish_state(data);
      }
    } else if (pkt.type == MeterBatch) {
      batch = reassemble_and_decode_batch(pkt);
      for (sample in batch) {
        update_sender_status(sample);
        mqtt_publish_state(sample);
      }
    }
  }

  if (time_to_send_timesync()) {
    time_send_timesync(self_short_id); // 60s for first 10 min, then hourly if RTC is present
  }

  mqtt_loop();
  web_server_loop();
  display_set_receiver_status(...);
  display_tick();
}
```

**Key receiver functions**:
```cpp
bool lora_receive(LoraPacket &pkt, uint32_t timeout_ms);
bool jsonToMeterData(const String &json, MeterData &data);
bool decode_batch(const uint8_t *buf, size_t len, BatchInput *out);
bool mqtt_publish_state(const MeterData &data);
void web_server_loop();                   // AP or STA UI
void time_send_timesync(uint16_t self_id);
```

## Test Mode (compile-time)
Enabled by `-DENABLE_TEST_MODE` (see `platformio.ini` test environment).

- Sender: sends 4-digit test code every ~30s in JSON.
- Receiver: shows last test code per sender and publishes to `/test` topic.
- Normal behavior is excluded from test builds.

**Test sender (pseudo-code)**:
```cpp
void test_sender_loop() {
  code = random_4_digits();
  json = {id, role:"sender", test_code: code, ts};
  lora_send(packet(TestCode, compress(json)));
  display_set_test_code(code);
}
```

**Test receiver (pseudo-code)**:
```cpp
void test_receiver_loop() {
  if (pkt.type == TestCode) {
    json = decompress(pkt.payload);
    update_sender_test_code(json);
    mqtt_publish_test(id, json);
  }
}
```

## LoRa Protocol
Packet layout:

```
[0]    protocol_version (1)
[1]    role (0=sender, 1=receiver)
[2..3] device_id_short (uint16)
[4]    payload_type (0=meter, 1=test, 2=time_sync, 3=meter_batch, 4=ack)
[5..N-3] payload bytes (compressed JSON for MeterData, binary for MeterBatch/Test/TimeSync)
[N-2..N-1] CRC16 (bytes 0..N-3)
```

LoRa radio settings:
- Frequency: **433 MHz** or **868 MHz** (set by build env via `LORA_FREQUENCY_HZ`)
- SF12, BW 125 kHz, CR 4/5, CRC on, Sync Word 0x34

## Data Format
MeterData JSON (sender + MQTT):

```json
{
  "id": "F19C",
  "ts": 1737200000,
  "e_kwh": 1234.57,
  "p_w": 950.00,
  "p1_w": 500.00,
  "p2_w": 450.00,
  "p3_w": 0.00,
  "bat_v": 3.92,
  "bat_pct": 78
}
```

### Binary MeterBatch Payload (LoRa)
Fixed header (little-endian):
- `magic` u16 = 0xDDB3
- `schema` u8 = 1
- `flags` u8 = 0x01 (bit0 = signed phases)
- `sender_id` u16 (1..NUM_SENDERS, maps to `EXPECTED_SENDER_IDS`)
- `batch_id` u16
- `t_last` u32 (unix seconds of last sample)
- `dt_s` u8 (seconds, >0)
- `n` u8 (sample count, <=30)
- `battery_mV` u16
- `err_m` u8 (meter read failures, sender-side counter)
- `err_d` u8 (decode failures, sender-side counter)
- `err_tx` u8 (LoRa TX failures, sender-side counter)
- `err_last` u8 (last error code: 0=None, 1=MeterRead, 2=Decode, 3=LoraTx)

Body:
- `E0` u32 (absolute energy in Wh)
- `dE[1..n-1]` ULEB128 (delta vs previous, >=0)
- `P1_0` s16 (absolute W)
- `dP1[1..n-1]` signed varint (ZigZag + ULEB128)
- `P2_0` s16
- `dP2[1..n-1]` signed varint
- `P3_0` s16
- `dP3[1..n-1]` signed varint

Notes:
- Receiver reconstructs timestamps from `t_last` and `dt_s`.
- Total power is computed on receiver as `p1 + p2 + p3`.
- Sender error counters are carried in the batch header and applied to all samples.

## Device IDs
- Derived from WiFi STA MAC.
- `short_id = (MAC[4] << 8) | MAC[5]`
- `device_id = dd3-%04X`
- JSON `id` uses only the last 4 hex digits (e.g., `F19C`) to save airtime.

Receiver expects known senders in `include/config.h` via:
```cpp
constexpr uint8_t NUM_SENDERS = 1;
inline constexpr uint16_t EXPECTED_SENDER_IDS[NUM_SENDERS] = { 0xF19C };
```

## OLED Behavior
- Sender: OLED stays **ON for 10 seconds** on each wake, then powers down for sleep.
- Receiver: OLED is always on (no auto-off).
- Pages rotate every 4s.

## Power & Battery
- Sender disables WiFi/BLE, reads VBAT via ADC, uses linear SoC map:
  - 3.0 V = 0%
  - 4.2 V = 100%
- Uses deep sleep between cycles (`SENDER_WAKE_INTERVAL_SEC`).
- Sender CPU is throttled to 80 MHz and LoRa RX is only enabled in short windows (ACK wait or time-sync).

## Web UI
- AP SSID: `DD3-Bridge-<short_id>` (prefix configurable)
- AP password: `changeme123` (configurable)
- Endpoints:
  - `/`: status overview
  - `/wifi`: WiFi/MQTT/NTP config (AP and STA)
  - `/sender/<device_id>`: per-sender details
- Sender IDs on `/` are clickable (open sender page in a new tab).
- In STA mode, the UI is also available via the board's IP/hostname on your WiFi network.
- Main page shows SD card file listing (downloadable).
- Sender page includes a history chart (power) with configurable range/resolution/mode.

## Security
- Basic Auth is supported for the web UI. In STA mode it is enabled by default; AP mode is optional.
- Config flags in `include/config.h`:
  - `WEB_AUTH_REQUIRE_STA` (default `true`)
  - `WEB_AUTH_REQUIRE_AP` (default `false`)
  - `WEB_AUTH_DEFAULT_USER` / `WEB_AUTH_DEFAULT_PASS`
- Web credentials are stored in NVS. `/wifi`, `/sd/download`, `/history/*`, `/`, `/sender/*`, and `/manual` require auth when enabled.
- Password inputs are not prefilled. Leaving a password blank keeps the stored value; use the "clear password" checkbox to erase it.
- User-controlled strings are HTML-escaped before embedding in pages.

## MQTT
- Topic: `smartmeter/<deviceId>/state`
- QoS 0
- Test mode: `smartmeter/<deviceId>/test`
- Client ID: `dd3-bridge-<device_id>` (stable, derived from MAC)

## NTP
- NTP servers are configurable in the web UI (`/wifi`).
- Defaults: `pool.ntp.org` and `time.nist.gov`.
## RTC (DS3231)
- Optional DS3231 on the I2C bus. Connect SDA to GPIO21 and SCL to GPIO22 (same bus as the OLED).
- Enable/disable with `ENABLE_DS3231` in `include/config.h`.
- Receiver time sync packets set the RTC.
- On boot, if no LoRa time sync has arrived yet, the sender uses the RTC time as the initial `ts_utc`.
- When no RTC is present or enabled, the receiver keeps sending time sync every 60 seconds.

## Build Environments
- `lilygo-t3-v1-6-1`: production build (debug on)
- `lilygo-t3-v1-6-1-test`: test build with `ENABLE_TEST_MODE`
- `lilygo-t3-v1-6-1-868`: production build for 868 MHz modules (debug on)
- `lilygo-t3-v1-6-1-868-test`: test build for 868 MHz modules
- `lilygo-t3-v1-6-1-payload-test`: build with `PAYLOAD_CODEC_TEST`
- `lilygo-t3-v1-6-1-868-payload-test`: 868 MHz build with `PAYLOAD_CODEC_TEST`
- `lilygo-t3-v1-6-1-prod`: production build with serial debug off
- `lilygo-t3-v1-6-1-868-prod`: 868 MHz production build with serial debug off

## Config Knobs
Key timing settings in `include/config.h`:
- `METER_SAMPLE_INTERVAL_MS`
- `METER_SEND_INTERVAL_MS`
- `BATCH_ACK_TIMEOUT_MS`
- `BATCH_MAX_RETRIES`
 - `BATCH_QUEUE_DEPTH`
 - `BATCH_RETRY_POLICY` (keep or drop on retry exhaustion)
 - `SERIAL_DEBUG_MODE_FLAG` (build flag) / `SERIAL_DEBUG_DUMP_JSON`
 - `LORA_SEND_BYPASS` (debug only)
 - `ENABLE_SD_LOGGING` / `PIN_SD_CS`
 - `SENDER_TIMESYNC_WINDOW_MS`
 - `SENDER_TIMESYNC_CHECK_SEC_FAST` / `SENDER_TIMESYNC_CHECK_SEC_SLOW`
 - `SD_HISTORY_MAX_DAYS` / `SD_HISTORY_MIN_RES_MIN`
 - `SD_HISTORY_MAX_BINS` / `SD_HISTORY_TIME_BUDGET_MS`
 - `WEB_AUTH_REQUIRE_STA` / `WEB_AUTH_REQUIRE_AP` / `WEB_AUTH_DEFAULT_USER` / `WEB_AUTH_DEFAULT_PASS`

## Limits & Known Constraints
- **Compression**: MeterData uses lightweight RLE (good for JSON but not optimal).
- **OBIS parsing**: supports IEC 62056-21 ASCII (Mode D); may need tuning for some meters.
- **Payload size**: single JSON frames < 256 bytes (ArduinoJson static doc); binary batch frames are chunked and reassembled (typically 1 chunk).
- **Battery ADC**: uses simple linear calibration constant in `power_manager.cpp`.
- **OLED**: no hardware reset line is used (matches working reference).
- **Batch ACKs**: sender waits for ACK after a batch and retries up to `BATCH_MAX_RETRIES` with `BATCH_ACK_TIMEOUT_MS` between attempts.

## SD Logging (Receiver)
Optional CSV logging to microSD (FAT32) when `ENABLE_SD_LOGGING = true`.

- Path: `/dd3/<device_id>/YYYY-MM-DD.csv`
- Columns:
  `ts_utc,p_w,p1_w,p2_w,p3_w,e_kwh,bat_v,bat_pct,rssi,snr,err_m,err_d,err_tx,err_last`
- `err_last` is written as text (`meter`, `decode`, `loratx`) only on the last sample of a batch that reports an error.
- Files are downloadable from the main UI page.
- Downloads only allow absolute paths under `/dd3/`, reject `..`, backslashes, and repeated slashes, and enforce a max path length.
- History chart on sender page stream-parses CSVs and bins data in the background.
- SD uses the on-board microSD SPI pins (CS=13, MOSI=15, SCK=14, MISO=2).

## Files & Modules
- `include/config.h`, `src/config.cpp`: pins, radio settings, sender IDs
- `include/data_model.h`, `src/data_model.cpp`: MeterData + ID init
- `include/json_codec.h`, `src/json_codec.cpp`: JSON encode/decode
- `include/compressor.h`, `src/compressor.cpp`: RLE compression
- `include/lora_transport.h`, `src/lora_transport.cpp`: LoRa packet + CRC
- `src/payload_codec.h`, `src/payload_codec.cpp`: binary batch encoder/decoder
- `include/meter_driver.h`, `src/meter_driver.cpp`: IEC 62056-21 ASCII parse
- `include/power_manager.h`, `src/power_manager.cpp`: ADC + sleep
- `include/time_manager.h`, `src/time_manager.cpp`: NTP + time sync
- `include/wifi_manager.h`, `src/wifi_manager.cpp`: NVS config + WiFi
- `include/mqtt_client.h`, `src/mqtt_client.cpp`: MQTT publish
- `include/web_server.h`, `src/web_server.cpp`: AP/STA web pages
- `include/display_ui.h`, `src/display_ui.cpp`: OLED pages + control
- `include/test_mode.h`, `src/test_mode.cpp`: test sender/receiver
- `src/main.cpp`: role detection and main loop

## Quick Start
1. Set role jumper on GPIO14:
   - LOW: sender
   - HIGH: receiver
2. OLED control on GPIO13:
   - HIGH: always on
   - LOW: auto-off after 10 minutes
3. Build and upload:

```bash
pio run -e lilygo-t3-v1-6-1 -t upload --upload-port COMx
```

Test mode:

```bash
pio run -e lilygo-t3-v1-6-1-test -t upload --upload-port COMx
```

868 MHz builds:

```bash
pio run -e lilygo-t3-v1-6-1-868 -t upload --upload-port COMx
```

868 MHz test mode:

```bash
pio run -e lilygo-t3-v1-6-1-868-test -t upload --upload-port COMx
```