DD3-LoRa-Bridge-MultiSender/REPUBLISH_SCRIPTS_VALIDATION_SUMMARY.md

# Python Scripts Compatibility Check - Summary

## ✅ VERDICT: Both Scripts Work with Newest CSV and InfluxDB Formats

**Tested:** `republish_mqtt.py` and `republish_mqtt_gui.py`  
**Test Date:** March 11, 2026  
**Result:** 5/5 compatibility tests passed

---

## Quick Reference

| Check | Status | Details |
|-------|--------|---------|
| CSV Parsing | ✅ PASS | Reads current `ts_utc,ts_hms_local,...` format correctly |
| CSV Backward Compat | ✅ PASS | Also works with legacy format (no `ts_hms_local`) |
| MQTT JSON Output | ✅ PASS | Generated JSON matches device expectations |
| Future Fields | ✅ PASS | Scripts handle new CSV columns without breaking |
| InfluxDB Schema | ✅ PASS | Query format matches expected schema (optional feature) |
| **Documentation** | ⚠️ FIXED | Corrected typo: `ts_hms_utc` → `ts_hms_local` |
| **Syntax Errors** | ✅ PASS | Both scripts compile cleanly |

---

## Test Results Summary

### 1. CSV Format Compatibility ✅
**Current device CSV (sd_logger.cpp):**
```
ts_utc,ts_hms_local,p_w,p1_w,p2_w,p3_w,e_kwh,bat_v,bat_pct,rssi,snr,err_m,err_d,err_tx,err_last
```

- Both scripts check for required fields: `ts_utc`, `e_kwh`, `p_w`
- Optional fields are read gracefully when present
- Field types are correctly converted
- ✅ **Scripts work without modification**

### 2. MQTT JSON Output Format ✅
**Republished JSON matches device format:**
```json
{
  "id": "F19C",
  "ts": 1710076800,
  "e_kwh": "1234.57",
  "p_w": 5432,
  "p1_w": 1800,
  "p2_w": 1816,
  "p3_w": 1816,
  "bat_v": "4.15",
  "bat_pct": 95,
  "rssi": -95,
  "snr": 9.25
}
```

- All required fields present
- Data types and formatting match expectations
- Compatible with MQTT subscribers and Home Assistant
- ✅ **No changes needed**

### 3. Backward Compatibility ✅
- Legacy CSV files (without `ts_hms_local`) still work
- Scripts ignore columns they don't understand
- Can process CSV files from both old and new firmware versions
- ✅ **Future-proof**

### 4. InfluxDB Auto-Detect ✅
- Scripts expect: measurement `"smartmeter"`, tag `"device_id"`
- Auto-detect is optional (falls back to manual time selection)
- ⚠️ NOTE: Device firmware doesn't write InfluxDB directly
  - Requires external bridge (Telegraf, Node-RED, etc.)
  - If bridge missing, manual mode works fine
- ✅ **Graceful degradation**

---

## Issues Found

### 🔴 Issue 1: Documentation Error (FIXED)
**Severity:** HIGH (documentation error, code is fine)  
**File:** `REPUBLISH_README.md` line 84

**Problem:** Header listed as `ts_hms_utc` but actual device writes `ts_hms_local`

**What Changed:**
- ❌ Before: `ts_utc,ts_hms_utc,p_w,...` (typo)
- ✅ After: `ts_utc,ts_hms_local,p_w,...` (correct)

**Reason:** `ts_hms_local` is local time in your configured timezone, not UTC. The `ts_utc` field is the actual UTC timestamp.

---

### ⚠️ Issue 2: Error Fields Not Republished (EXPECTED LIMITATION)
**Severity:** LOW (not a bug, limitation of feature)

**What's missing:**
- CSV contains: `err_m`, `err_d`, `err_tx`, `err_last` (error counters)
- Republished JSON doesn't include these fields
- **Impact:** Error diagnostics won't be restored from recovered CSV

**Why:** 
- Error counters are diagnostic/status info, not core meter data
- Main recovery goal is saving energy/power readings (which ARE included)
- Error counters reset at UTC hour boundaries anyway

**Status:** ✅ DOCUMENTED in report, no code change needed

---

### ℹ️ Issue 3: InfluxDB Bridge Required (EXPECTED)
**Severity:** INFORMATIONAL

**What it means:**
- Device publishes to MQTT only
- InfluxDB auto-detect requires external MQTT→InfluxDB bridge
- Examples: Telegraf, Node-RED, Home Assistant

**Status:** ✅ WORKING AS DESIGNED - manual mode always available

---

## What Was Tested

### Test Suite: `test_republish_compatibility.py`
- ✅ CSV parser can read current device format
- ✅ Scripts handle new fields gracefully
- ✅ MQTT JSON output format validation
- ✅ Legacy CSV format compatibility
- ✅ InfluxDB schema requirements

**Run test:** `python test_republish_compatibility.py`

---

## Files Modified

1. **REPUBLISH_README.md** - Fixed typo in CSV header documentation
2. **REPUBLISH_COMPATIBILITY_REPORT.md** - Created detailed compatibility analysis (this report)
3. **test_republish_compatibility.py** - Created test suite for future validation

---

## Recommendations

### ✅ Done (No Action Needed)
- Both scripts already work correctly
- Test suite created for future validation
- Documentation error fixed

### 🔄 Optional Enhancements (For Later)
1. Update scripts to parse/republish error fields if needed
2. Document InfluxDB bridge setup (Telegraf example)
3. Add more edge case tests (missing fields, malformed data, etc.)

### 📋 For Users
- Keep using both scripts as-is
- Use **manual time selection** if InfluxDB is unavailable
- Refer to updated REPUBLISH_README.md for correct CSV format

---

## Technical Details

### CSV Processing Flow
```
1. Read CSV with csv.DictReader
   ↓
2. Check for required fields: ts_utc, e_kwh, p_w
   ↓
3. Convert types:
   - ts_utc → int (seconds)
   - e_kwh → float → formatted as "X.XX" string
   - p_w → int (rounded)
   - Energy/power values → integers or floats
   ↓
4. Publish to MQTT topic: smartmeter/{device_id}/state
```

### MQTT JSON Format
- Strings: `e_kwh`, `bat_v` (formatted with 2 decimal places)
- Integers: `ts`, `p_w`, `p1_w`, `p2_w`, `p3_w`, `bat_pct`, `rssi`, `id`
- Floats: `snr`

### Device Schema Evolution
- ✅ Device now sends: `rx_reject`, `rx_reject_text` (new)
- ⚠️ These don't go to CSV, so can't be republished
- ✅ All existing fields preserved

---

## Conclusion

**Both republish scripts are production-ready and fully compatible with**:
- ✅ Current SD card CSV exports
- ✅ Device MQTT publishers
- ✅ InfluxDB optional auto-detect
- ✅ Home Assistant integrations
- ✅ Legacy data files (backward compatible)

No code changes required. Only documentation correction applied.