A hardware run exhausted the board's task/fd table after a few probe
cycles: shell open failed while heartbeat still worked, and continued
writes eventually stalled the link. Not a firmware hang, a resource leak
the tooling drove.
The firmware spawns a new nsh task plus two pipes (four fds) on the
first SERIAL_CONTROL write of a session (Mavlink::get_shell) and only
frees them when a SERIAL_CONTROL message arrives with the RESPOND flag
cleared (Mavlink::close_shell). Two tooling bugs combined to leak them:
- MavlinkShell.open() returned True on any buffered byte, including
leftover output from a prior session, so it reported a live shell
without one. It now confirms an actual 'nsh>' prompt or a completed
echo-sentinel round-trip before returning True, keeping the same
timeout-and-report-failure behavior.
- open/close cycles reopened before the single flags=0 teardown was
processed, spawning a second shell before the first was freed. close()
now sends the flags=0 teardown and then briefly drains incoming
SERIAL_CONTROL until quiet (bounded, never a hang) so the firmware
runs close_shell before a caller can reopen.
Lifecycle: every MavlinkShell user now opens one session, runs all its
work on it, and closes it in a finally so no error path leaks a shell.
The SIH flight and storage tests run all their probes on a single shell
(shell_command_exists already takes an open shell); flight_mission no
longer opens a throwaway probe shell separate from the flight session
beyond the unavoidable reboot boundary in enter_sih.
Folds in the sentinel fix (run() sends the echo sentinels on their own
input line, because nsh aborts a ';' chain when a command fails, which
previously lost the sentinel for a failing probe and read as a stall).
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
Turn the flat script pile into a standalone project so contributors can
navigate and extend it:
- px4bench/ shared library package: core primitives in __init__ (Reporter,
connect, MavlinkShell, reboot/replug, viewer tee, mavlink status parsers,
pymavlink add_message workaround) plus protocol modules params.py,
missions.py, and ftp.py extracted from the tests. Zero helper duplication
remains across scripts.
- bench/ holds the real-firmware tests (boot_health, reboot_loop,
usb_replug, link_forwarding, param_stress, mission_stress, log_transfer);
sih/ holds the simulated flight (flight_mission), making the
simulation/no-simulation boundary explicit.
- pyproject.toml (px4bench 0.1.0, BSD-3-Clause, pymavlink/pyserial deps,
pyulog extra) so pip install -e Tools/bench_test works; every script
remains directly runnable without installation via a parent-dir path shim.
- README rewritten contributor-first: architecture, per-test justification
tied to the v1.18 risk areas, why pymavlink over MAVSDK, how to add a
test, baseline workflow.
- Consistent CLI surface (shared connection args; --report-dir replaces
log_transfer's --outdir); decorative section banners removed.
All hardware-learned behavior is preserved exactly: param echo drain and
match-by-value, shell sentinel strip-all, the add_message workaround,
explicit param save before reboot, mission clear before upload, RTL in
MAV_FRAME_MISSION, and the post-flight ULog download.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
