Improve reliability and add data recovery tools
## Bug Fixes - Fix integer overflow potential in history bin allocation (web_server.cpp) Using uint64_t for intermediate multiplication prevents overflow with different constants - Prevent data loss during WiFi failures (main.cpp) Device now automatically attempts WiFi reconnection every 30 seconds when in AP mode Exits AP mode and resumes MQTT transmission as soon as WiFi becomes available Data collection and SD logging continue regardless of connectivity ## New Features - Add standalone MQTT data republisher for lost data recovery - Command-line tool (republish_mqtt.py) with interactive and scripting modes - GUI tool (republish_mqtt_gui.py) for user-friendly recovery - Rate-limited publishing (5 msg/sec default, configurable 1-100) - Manual time range selection or auto-detect missing data via InfluxDB - Cross-platform support (Windows, macOS, Linux) - Converts SD card CSV exports back to MQTT format ## Documentation - Add comprehensive code review (CODE_REVIEW.md) - 16 detailed security and quality assessments - Identifies critical HTTPS/auth gaps, medium-priority overflow issues - Confirms absence of buffer overflows and unsafe string functions - Grade: B+ with areas for improvement - Add republisher documentation (REPUBLISH_README.md, REPUBLISH_GUI_README.md) - Installation and usage instructions - Example commands and scenarios - Troubleshooting guide - Performance characteristics ## Dependencies - Add requirements_republish.txt - paho-mqtt>=1.6.1 - influxdb-client>=1.18.0 ## Impact - Eliminates data loss scenario where unreliable WiFi leaves device stuck in AP mode - Provides recovery mechanism for any historical data missed during outages - Improves code safety with explicit overflow-resistant arithmetic - Increases operational visibility with comprehensive code review
This commit is contained in:
181
REPUBLISH_GUI_README.md
Normal file
181
REPUBLISH_GUI_README.md
Normal file
@@ -0,0 +1,181 @@
|
||||
# DD3 MQTT Data Republisher - GUI Version
|
||||
|
||||
User-friendly graphical interface for recovering lost meter data from SD card CSV files and republishing to MQTT.
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
# Install dependencies (same as CLI version)
|
||||
pip install -r requirements_republish.txt
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### Launch the GUI
|
||||
|
||||
```bash
|
||||
# Windows
|
||||
python republish_mqtt_gui.py
|
||||
|
||||
# macOS/Linux
|
||||
python3 republish_mqtt_gui.py
|
||||
```
|
||||
|
||||
## Interface Overview
|
||||
|
||||
### Settings Tab
|
||||
Configure MQTT connection and data source:
|
||||
- **CSV File**: Browse and select the CSV file from your SD card
|
||||
- **Device ID**: Device identifier (e.g., `dd3-F19C`)
|
||||
- **MQTT Settings**: Broker address, port, username/password
|
||||
- **Publish Rate**: Messages per second (1-100, default: 5)
|
||||
- **Test Connection**: Verify MQTT broker is reachable
|
||||
|
||||
### Time Range Tab
|
||||
Choose how to select the time range to republish:
|
||||
|
||||
#### Manual Mode (Always Available)
|
||||
- Enter start and end dates/times
|
||||
- Example: Start `2026-03-01` at `00:00:00`, End `2026-03-05` at `23:59:59`
|
||||
- Useful when you know exactly what data is missing
|
||||
|
||||
#### Auto-Detect Mode (Requires InfluxDB)
|
||||
- Automatically finds gaps in your InfluxDB data
|
||||
- Connect to your InfluxDB instance
|
||||
- Script will identify the oldest missing data range
|
||||
- Republish that range automatically
|
||||
|
||||
### Progress Tab
|
||||
Real-time status during publishing:
|
||||
- **Progress Bar**: Visual indication of publishing status
|
||||
- **Statistics**: Count of published/skipped samples, current rate
|
||||
- **Log Output**: Detailed logging of all actions
|
||||
|
||||
## Step-by-step Example
|
||||
|
||||
1. **Prepare CSV File**
|
||||
- Extract CSV file from SD card
|
||||
- Example path: `D:\dd3-F19C\2026-03-09.csv`
|
||||
|
||||
2. **Launch GUI**
|
||||
```bash
|
||||
python republish_mqtt_gui.py
|
||||
```
|
||||
|
||||
3. **Settings Tab**
|
||||
- Click "Browse..." and select the CSV file
|
||||
- Enter Device ID: `dd3-F19C`
|
||||
- MQTT Broker: `192.168.1.100` (or your broker address)
|
||||
- Test connection to verify MQTT is working
|
||||
|
||||
4. **Time Range Tab**
|
||||
- **Manual Mode**: Enter dates you want to republish
|
||||
- Start: `2026-03-09` / `08:00:00`
|
||||
- End: `2026-03-09` / `18:00:00`
|
||||
- **Or Auto-Detect**: Fill InfluxDB settings if available
|
||||
|
||||
5. **Progress Tab**
|
||||
- View real-time publishing progress
|
||||
- Watch the log for detailed status
|
||||
|
||||
6. **Start**
|
||||
- Click "Start Publishing" button
|
||||
- Monitor progress in real-time
|
||||
- Success message when complete
|
||||
|
||||
## Tips
|
||||
|
||||
### CSV File Location
|
||||
On Windows with SD card reader:
|
||||
- Drive letter shows up (e.g., `D:\`)
|
||||
- Path is usually: `D:\dd3\[DEVICE-ID]\[DATE].csv`
|
||||
|
||||
On Linux with SD card:
|
||||
- Example: `/mnt/sd/dd3/dd3-F19C/2026-03-09.csv`
|
||||
|
||||
### Finding Device ID
|
||||
- Displayed on device's OLED screen
|
||||
- Also in CSV directory names on SD card
|
||||
- Format: `dd3-XXXX` where XXXX is hex device short ID
|
||||
|
||||
### Rate Limiting
|
||||
- **Conservative** (1-2 msg/sec): For unreliable networks or busy brokers
|
||||
- **Default** (5 msg/sec): Recommended, safe for most setups
|
||||
- **Fast** (10+ msg/sec): Only if you know your broker can handle it
|
||||
|
||||
### InfluxDB Auto-Detect
|
||||
Requires:
|
||||
- InfluxDB running and accessible
|
||||
- Valid API token
|
||||
- Correct organization and bucket names
|
||||
- Data already stored in InfluxDB bucket
|
||||
|
||||
If InfluxDB unavailable: Fall back to manual time selection
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Could not connect to MQTT broker"
|
||||
- Check broker address and port
|
||||
- Verify firewall allows connection
|
||||
- Check if broker is running
|
||||
- Try "Test Connection" button
|
||||
|
||||
### "CSV file not found"
|
||||
- Verify file path is correct
|
||||
- Try re-selecting file with Browse button
|
||||
- Ensure file is readable
|
||||
|
||||
### "0 samples published"
|
||||
- Time range may not match CSV data
|
||||
- Try wider time range
|
||||
- Check CSV file contains data
|
||||
- Verify timestamps are Unix format
|
||||
|
||||
### "InfluxDB connection error"
|
||||
- Check InfluxDB URL is running
|
||||
- Verify API token is valid
|
||||
- Check organization and bucket name
|
||||
- Try accessing InfluxDB web UI manually
|
||||
|
||||
### GUI is slow or unresponsive
|
||||
- This is normal during MQTT publishing
|
||||
- GUI updates in background
|
||||
- Wait for operation to complete
|
||||
- Check Progress tab for live updates
|
||||
|
||||
## Keyboard Shortcuts
|
||||
- Tab: Move to next field
|
||||
- Enter: Start publishing from most tabs
|
||||
- Ctrl+C: Exit (if launched from terminal)
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
republish_mqtt.py → Command-line version
|
||||
republish_mqtt_gui.py → GUI version (this)
|
||||
requirements_republish.txt → Python dependencies
|
||||
REPUBLISH_README.md → Full documentation
|
||||
```
|
||||
|
||||
Use the **GUI** if you prefer point-and-click interface.
|
||||
Use the **CLI** if you want to automate or run in scripts.
|
||||
|
||||
## Platform Support
|
||||
|
||||
✓ **Windows 10/11** - Native support
|
||||
✓ **macOS** - Works with Python 3.7+
|
||||
✓ **Linux** (Ubuntu, Debian, Fedora) - Works with Python 3.7+
|
||||
|
||||
All platforms use tkinter (included with Python).
|
||||
|
||||
## Performance
|
||||
|
||||
Typical times on a standard PC:
|
||||
- 1 day of data (~2800 samples): ~9-10 minutes at 5 msg/sec
|
||||
- 1 week of data (~19,600 samples): ~65 minutes at 5 msg/sec
|
||||
|
||||
Time = (Number of Samples) / (Rate in msg/sec)
|
||||
|
||||
## License
|
||||
|
||||
Same as DD3 project
|
||||
Reference in New Issue
Block a user