DD3-LoRa-Bridge-MultiSender/REPUBLISH_SCRIPTS_VALIDATION_SUMMARY.md

Python Scripts Compatibility Check - Summary

✅ VERDICT: Both Scripts Work with Newest CSV and InfluxDB Formats

Tested: republish_mqtt.py and republish_mqtt_gui.py
Test Date: March 11, 2026
Result: 5/5 compatibility tests passed

---

Quick Reference

Check	Status	Details
CSV Parsing	✅ PASS	Reads current ts_utc,ts_hms_local,... format correctly
CSV Backward Compat	✅ PASS	Also works with legacy format (no ts_hms_local)
MQTT JSON Output	✅ PASS	Generated JSON matches device expectations
Future Fields	✅ PASS	Scripts handle new CSV columns without breaking
InfluxDB Schema	✅ PASS	Query format matches expected schema (optional feature)
Documentation	⚠️ FIXED	Corrected typo: ts_hms_utc → ts_hms_local
Syntax Errors	✅ PASS	Both scripts compile cleanly

---

Test Results Summary

1. CSV Format Compatibility ✅

Current device CSV (sd_logger.cpp):

ts_utc,ts_hms_local,p_w,p1_w,p2_w,p3_w,e_kwh,bat_v,bat_pct,rssi,snr,err_m,err_d,err_tx,err_last

• Both scripts check for required fields: ts_utc, e_kwh, p_w
• Optional fields are read gracefully when present
• Field types are correctly converted
• ✅ Scripts work without modification

2. MQTT JSON Output Format ✅

Republished JSON matches device format:

{
  "id": "F19C",
  "ts": 1710076800,
  "e_kwh": "1234.57",
  "p_w": 5432,
  "p1_w": 1800,
  "p2_w": 1816,
  "p3_w": 1816,
  "bat_v": "4.15",
  "bat_pct": 95,
  "rssi": -95,
  "snr": 9.25
}

• All required fields present
• Data types and formatting match expectations
• Compatible with MQTT subscribers and Home Assistant
• ✅ No changes needed

3. Backward Compatibility ✅

• Legacy CSV files (without ts_hms_local) still work
• Scripts ignore columns they don't understand
• Can process CSV files from both old and new firmware versions
• ✅ Future-proof

4. InfluxDB Auto-Detect ✅

• Scripts expect: measurement "smartmeter", tag "device_id"
• Auto-detect is optional (falls back to manual time selection)
• ⚠️ NOTE: Device firmware doesn't write InfluxDB directly
  • Requires external bridge (Telegraf, Node-RED, etc.)
  • If bridge missing, manual mode works fine
• ✅ Graceful degradation

---

Issues Found

🔴 Issue 1: Documentation Error (FIXED)

Severity: HIGH (documentation error, code is fine)
File: REPUBLISH_README.md line 84

Problem: Header listed as ts_hms_utc but actual device writes ts_hms_local

What Changed:

• ❌ Before: ts_utc,ts_hms_utc,p_w,... (typo)
• ✅ After: ts_utc,ts_hms_local,p_w,... (correct)

Reason: ts_hms_local is local time in your configured timezone, not UTC. The ts_utc field is the actual UTC timestamp.

---

⚠️ Issue 2: Error Fields Not Republished (EXPECTED LIMITATION)

Severity: LOW (not a bug, limitation of feature)

What's missing:

• CSV contains: err_m, err_d, err_tx, err_last (error counters)
• Republished JSON doesn't include these fields
• Impact: Error diagnostics won't be restored from recovered CSV

Why:

• Error counters are diagnostic/status info, not core meter data
• Main recovery goal is saving energy/power readings (which ARE included)
• Error counters reset at UTC hour boundaries anyway

Status: ✅ DOCUMENTED in report, no code change needed

---

ℹ️ Issue 3: InfluxDB Bridge Required (EXPECTED)

Severity: INFORMATIONAL

What it means:

• Device publishes to MQTT only
• InfluxDB auto-detect requires external MQTT→InfluxDB bridge
• Examples: Telegraf, Node-RED, Home Assistant

Status: ✅ WORKING AS DESIGNED - manual mode always available

---

What Was Tested

Test Suite: test_republish_compatibility.py

• ✅ CSV parser can read current device format
• ✅ Scripts handle new fields gracefully
• ✅ MQTT JSON output format validation
• ✅ Legacy CSV format compatibility
• ✅ InfluxDB schema requirements

Run test: python test_republish_compatibility.py

---

Files Modified

1. REPUBLISH_README.md - Fixed typo in CSV header documentation
2. REPUBLISH_COMPATIBILITY_REPORT.md - Created detailed compatibility analysis (this report)
3. test_republish_compatibility.py - Created test suite for future validation

---

Recommendations

✅ Done (No Action Needed)

• Both scripts already work correctly
• Test suite created for future validation
• Documentation error fixed

🔄 Optional Enhancements (For Later)

1. Update scripts to parse/republish error fields if needed
2. Document InfluxDB bridge setup (Telegraf example)
3. Add more edge case tests (missing fields, malformed data, etc.)

📋 For Users

• Keep using both scripts as-is
• Use manual time selection if InfluxDB is unavailable
• Refer to updated REPUBLISH_README.md for correct CSV format

---

Technical Details

CSV Processing Flow

1. Read CSV with csv.DictReader
   ↓
2. Check for required fields: ts_utc, e_kwh, p_w
   ↓
3. Convert types:
   - ts_utc → int (seconds)
   - e_kwh → float → formatted as "X.XX" string
   - p_w → int (rounded)
   - Energy/power values → integers or floats
   ↓
4. Publish to MQTT topic: smartmeter/{device_id}/state

MQTT JSON Format

• Strings: e_kwh, bat_v (formatted with 2 decimal places)
• Integers: ts, p_w, p1_w, p2_w, p3_w, bat_pct, rssi, id
• Floats: snr

Device Schema Evolution

• ✅ Device now sends: rx_reject, rx_reject_text (new)
• ⚠️ These don't go to CSV, so can't be republished
• ✅ All existing fields preserved

---

Conclusion

Both republish scripts are production-ready and fully compatible with:

• ✅ Current SD card CSV exports
• ✅ Device MQTT publishers
• ✅ InfluxDB optional auto-detect
• ✅ Home Assistant integrations
• ✅ Legacy data files (backward compatible)

No code changes required. Only documentation correction applied.