89 lines
6.4 KiB
Markdown
89 lines
6.4 KiB
Markdown
# Test Matrix
|
|
|
|
Use `python tools/verify.py` as the canonical non-hardware verification command.
|
|
|
|
## Non-Hardware Verification
|
|
|
|
| Check | Command | Hardware | Required in CI |
|
|
|---|---|---|---|
|
|
| Repository, dependency, and PlatformIO policy | `python tools/check_dependencies.py` | None | Yes |
|
|
| Text hygiene and Python syntax | `python tools/check_format.py` | None | Yes |
|
|
| Production build | `python -m platformio run -e production` | None | Yes |
|
|
| Debug build | `python -m platformio run -e debug` | None | Yes |
|
|
| Test firmware build | `python -m platformio run -e test` | None | Yes |
|
|
| Compile-only Unity suites | `python -m platformio test -e test --without-uploading --without-testing` | None | Yes |
|
|
| Static analysis | `python -m platformio check -e production --fail-on-defect=high` | None | Yes |
|
|
| Documentation consistency | `python tools/check_docs.py` | None | Yes |
|
|
| Firmware size report | `python tools/check_size.py` | None after builds | Yes |
|
|
|
|
## Existing Embedded Unity Suites
|
|
|
|
| Suite | Primary coverage | Compile-only command | On-device command |
|
|
|---|---|---|---|
|
|
| `test_payload_codec` | Payload schema v4, sparse mask, varints, golden vectors | `python -m platformio test -e test --without-uploading --without-testing -f test_payload_codec` | `python -m platformio test -e test -f test_payload_codec` |
|
|
| `test_lora_transport` | CRC16, LoRa frame build/parse, chunk reassembly | `python -m platformio test -e test --without-uploading --without-testing -f test_lora_transport` | `python -m platformio test -e test -f test_lora_transport` |
|
|
| `test_json_codec` | MQTT JSON keys, optional keys, HA discovery fields | `python -m platformio test -e test --without-uploading --without-testing -f test_json_codec` | `python -m platformio test -e test -f test_json_codec` |
|
|
| `test_html_escape` | HTML escaping, URL encoding, web device-ID validation | `python -m platformio test -e test --without-uploading --without-testing -f test_html_escape` | `python -m platformio test -e test -f test_html_escape` |
|
|
| `test_refactor_smoke` | Public refactor headers and HA manufacturer guard | `python -m platformio test -e test --without-uploading --without-testing -f test_refactor_smoke` | `python -m platformio test -e test -f test_refactor_smoke` |
|
|
| `test_meter_fault_count` | Stale-meter fault counter regression | `python -m platformio test -e test --without-uploading --without-testing -f test_meter_fault_count` | `python -m platformio test -e test -f test_meter_fault_count` |
|
|
| `test_security_fuzz` | Malformed payload/frame/reassembly rejection | `python -m platformio test -e test --without-uploading --without-testing -f test_security_fuzz` | `python -m platformio test -e test -f test_security_fuzz` |
|
|
|
|
## Hardware Requirements
|
|
|
|
| Scenario | Boards | Smart meter | Wi-Fi | MQTT | SD |
|
|
|---|---:|---:|---:|---:|---:|
|
|
| Compile-only Unity tests | 0 | No | No | No | No |
|
|
| On-device Unity test suite | 1 | No | No | No | No |
|
|
| Test-mode sender or receiver smoke | 1 | No | Optional | Optional | Optional |
|
|
| Normal sender boot and sampling | 1 sender | Yes or simulator | No | No | No |
|
|
| Normal receiver boot and web/AP fallback | 1 receiver | No | Optional | Optional | Optional |
|
|
| End-to-end batch transfer | 1 sender + 1 receiver | Yes or simulator | Yes | Yes | Optional |
|
|
| SD logging and history | 1 receiver plus sender traffic | Optional | Optional | Optional | Yes |
|
|
| Power measurement | 1 sender plus receiver ACK source | Yes or simulator | No | No | No |
|
|
|
|
## Contract Traceability
|
|
|
|
| Contract | Tests or checks |
|
|
|---|---|
|
|
| LoRa frame format | `test_lora_transport`, `test_security_fuzz`, `tools/check_docs.py` |
|
|
| CRC validation | `test_lora_transport`, `test_security_fuzz` |
|
|
| Chunk reassembly | `test_lora_transport`, `test_security_fuzz` |
|
|
| Payload schema | `test_payload_codec`, `test_security_fuzz`, `tools/check_docs.py` |
|
|
| Sender-ID validation | Compile coverage in `test_refactor_smoke`; runtime behavior documented in `README.md`; HIL planned |
|
|
| ACK matching | Runtime behavior documented in `README.md`; HIL planned |
|
|
| Time bootstrap | Payload sync vector in `test_payload_codec`; runtime behavior documented in `README.md`; HIL planned |
|
|
| Retry bounds | Constants and state machine compile coverage; `test_meter_fault_count` covers one retry-adjacent regression; extraction planned |
|
|
| Queue bounds | State machine compile coverage; extraction planned |
|
|
| Duplicate handling | Runtime behavior documented in `README.md`; HIL planned |
|
|
| MQTT JSON keys | `test_json_codec` |
|
|
| Home Assistant discovery fields | `test_json_codec`, `test_refactor_smoke`, `test/check_ha_manufacturer.ps1` |
|
|
| CSV compatibility | Documented in `README.md` and `docs/engineering/current-state.md`; parser extraction planned |
|
|
| HTML escaping | `test_html_escape` |
|
|
| Web input validation | `test_html_escape`; SD/history route validation extraction planned |
|
|
|
|
## Native-Test Feasibility
|
|
|
|
A PlatformIO `native` environment should not be introduced until pure logic is separated from Arduino headers. The best first candidates are already close to pure logic:
|
|
|
|
- CRC functions and LoRa frame build/parse in `lib/dd3_transport_logic`.
|
|
- Batch reassembly in `lib/dd3_transport_logic`.
|
|
- Payload encoding/decoding and varints in `lib/dd3_legacy_core`.
|
|
- HTML escaping, URL encoding, and device-ID sanitization in `lib/dd3_legacy_core`.
|
|
- HA discovery JSON builder once `ArduinoJson` and `String` dependencies are handled.
|
|
|
|
Current blockers:
|
|
|
|
- Public helper headers include `Arduino.h`.
|
|
- Several otherwise pure helpers expose Arduino `String`.
|
|
- ArduinoJson is used directly in host-interesting code.
|
|
- Runtime logic depends on global Arduino services: `millis`, `delay`, `Serial`, `Serial2`, `LoRa`, `WiFi`, `SD`, `Preferences`, and ESP32 FreeRTOS APIs.
|
|
- Sender state-transition logic is still coupled to module-static globals and hardware-facing helpers.
|
|
|
|
Staged extraction plan:
|
|
|
|
1. Remove unnecessary `Arduino.h` includes from pure helper headers by replacing Arduino integer types with standard C/C++ headers.
|
|
2. Add native tests for `lora_frame_logic` and `batch_reassembly_logic`.
|
|
3. Split payload codec into a header/source pair that only needs standard C++ types, then add native golden-vector tests.
|
|
4. Add small pure helpers for ACK parse/validation, sparse-mask timestamp reconstruction, retry timeout calculation, and CSV line parsing.
|
|
5. Keep Arduino wrappers in firmware modules and test extracted helpers natively plus embedded compile-only suites.
|