The full-suite persistence check failed on hardware with 'after reboot
SDLOG_UTC_OFFSET = 740 (expected 777)'. Firmware persistence is solid;
this was a test race. A PARAM_SET propagates through the param system
asynchronously, and phase_persistence ran 'param save' on the strength
of the PARAM_VALUE echo alone. When save raced ahead of the commit it
persisted the PRIOR value (740, the last value Phase 2 wrote). The echo
was not even proof of the new value: wait_param_echo returned
seen=[740, 777], a stale queued PARAM_VALUE from Phase 2 having slipped
past drain_param_values.
Make persistence deterministic with two gates instead of timing:
1. After setting the marker, read it back from the board until it reports
MARKER_VALUE (read_until) before saving. This confirms the value is
committed to RAM and trusts the board's read over any stale echo.
2. After 'param save', confirm the saved state with 'param show <name>'
and require the '+' (saved) flag AND the marker value before rebooting
(retry save once first). '*' means unsaved; the test never reboots on
an unsaved marker. Flag columns per src/systemcmds/param/param.cpp:822
(x used, + saved, * unsaved, l locked).
phase_set_readback now also trusts read_until over the echo, so the same
stale-queued-echo does not fail it either; its pass criteria stay the
readback match. Restores in phase_persistence and the final cleanup use
read_until too.
New shared helpers in px4bench.params: read_until(mav, name, expected,
timeout) -> (ok, last_seen), and param_is_saved(shell, name) plus its
parse_param_show() parser for the 'param show' saved flag and value.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
A hardware run exhausted the board's task/fd table after a few probe
cycles: shell open failed while heartbeat still worked, and continued
writes eventually stalled the link. Not a firmware hang, a resource leak
the tooling drove.
The firmware spawns a new nsh task plus two pipes (four fds) on the
first SERIAL_CONTROL write of a session (Mavlink::get_shell) and only
frees them when a SERIAL_CONTROL message arrives with the RESPOND flag
cleared (Mavlink::close_shell). Two tooling bugs combined to leak them:
- MavlinkShell.open() returned True on any buffered byte, including
leftover output from a prior session, so it reported a live shell
without one. It now confirms an actual 'nsh>' prompt or a completed
echo-sentinel round-trip before returning True, keeping the same
timeout-and-report-failure behavior.
- open/close cycles reopened before the single flags=0 teardown was
processed, spawning a second shell before the first was freed. close()
now sends the flags=0 teardown and then briefly drains incoming
SERIAL_CONTROL until quiet (bounded, never a hang) so the firmware
runs close_shell before a caller can reopen.
Lifecycle: every MavlinkShell user now opens one session, runs all its
work on it, and closes it in a finally so no error path leaks a shell.
The SIH flight and storage tests run all their probes on a single shell
(shell_command_exists already takes an open shell); flight_mission no
longer opens a throwaway probe shell separate from the flight session
beyond the unavoidable reboot boundary in enter_sih.
Folds in the sentinel fix (run() sends the echo sentinels on their own
input line, because nsh aborts a ';' chain when a command fails, which
previously lost the sentinel for a failing probe and read as a stall).
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
The firmware already ships sd_bench, sd_stress, and serial_test; the
suite was not exploiting them.
bench/storage_stress.py joins the default sequence before log_transfer
(storage health first, then the test that depends on storage). It runs
sd_bench with data verification and parses write/read throughput and
fsync latency into named checks with generous FAIL floors (defaults
justified in the README: 100 KB/s write, 200 KB/s read, 500 ms fsync;
a working-but-slow card prints a WARNING instead of flapping), then
sd_stress for file churn with a short default iteration count. Both
phases probe-and-skip independently: a firmware without a command or a
board without an SD card records SKIP with a warning, never FAIL.
bench/serial_loopback.py is a standalone operator/fixture test for
manufacturing: with a TX-RX loopback jumper installed on a chosen UART,
serial_test transmits a known pattern and the session summary
(rx/tx/rx err counts) plus pattern-error lines decide the verdict. Not
in the default sequence because it needs a physical fixture.
The firmware gate's --build path now prints a bench capability report
from one shared table: after reading default.px4board plus the label
overlay (labels merge both, cmake/kconfig.cmake:47-54) it lists whether
the image will contain simulator_sih, sd_bench, sd_stress, and
serial_test and which suite tests will run or skip. For anything
missing it names a sibling variant that has it (exact --target) and
the exact config line, and interactively offers to append the line to
the local board config, stating that the edit is local and uncommitted.
All four options are 'default n' in Kconfig, so the merged .px4board
lines decide presence.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
Turn the flat script pile into a standalone project so contributors can
navigate and extend it:
- px4bench/ shared library package: core primitives in __init__ (Reporter,
connect, MavlinkShell, reboot/replug, viewer tee, mavlink status parsers,
pymavlink add_message workaround) plus protocol modules params.py,
missions.py, and ftp.py extracted from the tests. Zero helper duplication
remains across scripts.
- bench/ holds the real-firmware tests (boot_health, reboot_loop,
usb_replug, link_forwarding, param_stress, mission_stress, log_transfer);
sih/ holds the simulated flight (flight_mission), making the
simulation/no-simulation boundary explicit.
- pyproject.toml (px4bench 0.1.0, BSD-3-Clause, pymavlink/pyserial deps,
pyulog extra) so pip install -e Tools/bench_test works; every script
remains directly runnable without installation via a parent-dir path shim.
- README rewritten contributor-first: architecture, per-test justification
tied to the v1.18 risk areas, why pymavlink over MAVSDK, how to add a
test, baseline workflow.
- Consistent CLI surface (shared connection args; --report-dir replaces
log_transfer's --outdir); decorative section banners removed.
All hardware-learned behavior is preserved exactly: param echo drain and
match-by-value, shell sentinel strip-all, the add_message workaround,
explicit param save before reboot, mission clear before upload, RTL in
MAV_FRAME_MISSION, and the post-flight ULog download.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
