Update README for binary batch payload and SF11

README.md

@@ -1,6 +1,6 @@
 # DD3 LoRa Bridge (Multi-Sender)
 
-Unified firmware for LilyGO T3 v1.6.1 (ESP32 + SX1276 + SSD1306) that runs as **Sender** or **Receiver** based on a GPIO jumper. Senders read DD3 smart meter values, compress JSON, and transmit over LoRa. The receiver validates packets, publishes to MQTT, provides a web UI, and shows per-sender status on the OLED.
+Unified firmware for LilyGO T3 v1.6.1 (ESP32 + SX1276 + SSD1306) that runs as **Sender** or **Receiver** based on a GPIO jumper. Senders read DD3 smart meter values and transmit compact binary batches over LoRa. The receiver validates packets, publishes to MQTT, provides a web UI, and shows per-sender status on the OLED.
 
 ## Hardware
 Board: **LilyGO T3 LoRa32 v1.6.1** (ESP32 + SX1276 + SSD1306 128x64 + LiPo)
@@ -46,7 +46,7 @@ Variants:
   - Total power: 1-0:16.7.0*255
   - Phase power: 36.7 / 56.7 / 76.7
 - Reads battery voltage and estimates SoC.
-- Builds JSON payload, compresses, wraps in LoRa packet, transmits.
+- Builds compact binary batch payload, wraps in LoRa packet, transmits.
 - Light sleeps between meter reads; batches are sent every 30s.
 - Listens for LoRa time sync packets to set UTC clock.
 - Uses DS3231 RTC after boot if no time sync has arrived yet.
@@ -59,9 +59,8 @@ void sender_loop() {
   read_battery(data);        // VBAT + SoC
 
   if (time_to_send_batch()) {
-    json = meterBatchToJson(samples, batch_id); // bat_v per batch, t_first/t_last included
+    payload = encode_batch(samples, batch_id); // compact binary batch
-    compressed = compressBuffer(json);
+    lora_send(packet(MeterBatch, payload));
-    lora_send(packet(MeterBatch, compressed));
   }
 
   display_set_last_meter(data);
@@ -79,14 +78,14 @@ void sender_loop() {
 bool meter_read(MeterData &data);        // parse OBIS fields
 void read_battery(MeterData &data);      // ADC -> volts + percent
 bool meterDataToJson(const MeterData&, String&);
-bool compressBuffer(const uint8_t*, size_t, uint8_t*, size_t, size_t&);
+bool compressBuffer(const uint8_t*, size_t, uint8_t*, size_t, size_t&); // MeterData only
 bool lora_send(const LoraPacket &pkt);   // add header + CRC16 and transmit
 ```
 
 ### Receiver (USB-powered)
 - WiFi STA connect using stored config; if not available/fails, starts AP.
 - NTP sync (UTC) and local display in Europe/Berlin.
-- Receives LoRa packets, verifies CRC16, decompresses, parses JSON.
+- Receives LoRa packets, verifies CRC16, decompresses MeterData JSON, decodes binary batches.
 - Publishes meter JSON to MQTT.
 - Sends ACKs for MeterBatch packets and de-duplicates by batch_id.
 - Web UI:
@@ -105,8 +104,8 @@ void receiver_loop() {
         mqtt_publish_state(data);
       }
     } else if (pkt.type == MeterBatch) {
-      json = reassemble_and_decompress_batch(pkt);
+      batch = reassemble_and_decode_batch(pkt);
-      for (sample in jsonToMeterBatch(json)) { // uses t_first/t_last for jittered timestamps
+      for (sample in batch) {
         update_sender_status(sample);
         mqtt_publish_state(sample);
       }
@@ -128,7 +127,7 @@ void receiver_loop() {
 ```cpp
 bool lora_receive(LoraPacket &pkt, uint32_t timeout_ms);
 bool jsonToMeterData(const String &json, MeterData &data);
-bool jsonToMeterBatch(const String &json, MeterData *samples, size_t max, size_t &count);
+bool decode_batch(const uint8_t *buf, size_t len, BatchInput *out);
 bool mqtt_publish_state(const MeterData &data);
 void web_server_loop();                   // AP or STA UI
 void time_send_timesync(uint16_t self_id);
@@ -170,16 +169,16 @@ Packet layout:
 [1]    role (0=sender, 1=receiver)
 [2..3] device_id_short (uint16)
 [4]    payload_type (0=meter, 1=test, 2=time_sync, 3=meter_batch, 4=ack)
-[5..N-3] compressed payload
+[5..N-3] payload bytes (compressed JSON for MeterData, binary for MeterBatch/Test/TimeSync)
 [N-2..N-1] CRC16 (bytes 0..N-3)
 ```
 
 LoRa radio settings:
 - Frequency: **433 MHz** or **868 MHz** (set by build env via `LORA_FREQUENCY_HZ`)
-- SF10, BW 125 kHz, CR 4/5, CRC on, Sync Word 0x34
+- SF11, BW 125 kHz, CR 4/5, CRC on, Sync Word 0x34
 
 ## Data Format
-JSON payload (sender + MQTT):
+MeterData JSON (sender + MQTT):
 
 ```json
 {
@@ -195,36 +194,31 @@ JSON payload (sender + MQTT):
 }
 ```
 
-MeterBatch JSON (compressed over LoRa) uses per-field arrays with integer units for easier ingestion:
+### Binary MeterBatch Payload (LoRa)
+Fixed header (little-endian):
+- `magic` u16 = 0xDDB3
+- `schema` u8 = 1
+- `flags` u8 = 0x01 (bit0 = signed phases)
+- `sender_id` u16 (1..NUM_SENDERS, maps to `EXPECTED_SENDER_IDS`)
+- `batch_id` u16
+- `t_last` u32 (unix seconds of last sample)
+- `dt_s` u8 (seconds, >0)
+- `n` u8 (sample count, <=30)
+- `battery_mV` u16
 
-```json
+Body:
-{
+- `E0` u32 (absolute energy in Wh)
-  "schema": 1,
+- `dE[1..n-1]` ULEB128 (delta vs previous, >=0)
-  "sender": "s01",
+- `P1_0` s16 (absolute W)
-  "batch_id": 1842,
+- `dP1[1..n-1]` signed varint (ZigZag + ULEB128)
-  "t0": 1738288000,
+- `P2_0` s16
-  "t_first": 1738288000,
+- `dP2[1..n-1]` signed varint
-  "t_last": 1738288030,
+- `P3_0` s16
-  "dt_s": 1,
+- `dP3[1..n-1]` signed varint
-  "n": 3,
-  "e_wh": [123456700, 123456701, 123456701],
-  "p_w": [930, 940, 950],
-  "p1_w": [480, 490, 500],
-  "p2_w": [450, 450, 450],
-  "p3_w": [0, 0, 0],
-  "bat_v": 3.92,
-  "meta": {
-    "rssi": -92,
-    "snr": 7.5,
-    "rx_ts": 1738288031
-  }
-}
-```
 
 Notes:
-- `sender` maps to `EXPECTED_SENDER_IDS` order (`s01` = first sender).
+- Receiver reconstructs timestamps from `t_last` and `dt_s`.
-- `meta` is injected by the receiver after batch reassembly.
+- Total power is computed on receiver as `p1 + p2 + p3`.
-- `bat_v` is a single batch-level value (percent is calculated locally).
 
 ## Device IDs
 - Derived from WiFi STA MAC.
@@ -291,9 +285,9 @@ Key timing settings in `include/config.h`:
  - `LORA_SEND_BYPASS` (debug only)
 
 ## Limits & Known Constraints
-- **Compression**: uses lightweight RLE (good for JSON but not optimal).
+- **Compression**: MeterData uses lightweight RLE (good for JSON but not optimal).
 - **OBIS parsing**: supports IEC 62056-21 ASCII (Mode D); may need tuning for some meters.
-- **Payload size**: single JSON frames < 256 bytes (ArduinoJson static doc); batch frames are chunked and reassembled.
+- **Payload size**: single JSON frames < 256 bytes (ArduinoJson static doc); binary batch frames are chunked and reassembled (typically 1 chunk).
 - **Battery ADC**: uses simple linear calibration constant in `power_manager.cpp`.
 - **OLED**: no hardware reset line is used (matches working reference).
 - **Batch ACKs**: sender waits for ACK after a batch and retries up to `BATCH_MAX_RETRIES` with `BATCH_ACK_TIMEOUT_MS` between attempts.
@@ -304,6 +298,7 @@ Key timing settings in `include/config.h`:
 - `include/json_codec.h`, `src/json_codec.cpp`: JSON encode/decode
 - `include/compressor.h`, `src/compressor.cpp`: RLE compression
 - `include/lora_transport.h`, `src/lora_transport.cpp`: LoRa packet + CRC
+- `src/payload_codec.h`, `src/payload_codec.cpp`: binary batch encoder/decoder
 - `include/meter_driver.h`, `src/meter_driver.cpp`: IEC 62056-21 ASCII parse
 - `include/power_manager.h`, `src/power_manager.cpp`: ADC + sleep
 - `include/time_manager.h`, `src/time_manager.cpp`: NTP + time sync

include/config.h

@@ -49,7 +49,7 @@ constexpr uint8_t PIN_METER_RX = 34;
 #define LORA_FREQUENCY_HZ 433E6
 #endif
 constexpr long LORA_FREQUENCY = LORA_FREQUENCY_HZ;
-constexpr uint8_t LORA_SPREADING_FACTOR = 10;
+constexpr uint8_t LORA_SPREADING_FACTOR = 11;
 constexpr long LORA_BANDWIDTH = 125E3;
 constexpr uint8_t LORA_CODING_RATE = 5;
 constexpr uint8_t LORA_SYNC_WORD = 0x34;