Some boards heartbeat from more than one component on the same sysid. An
FMUv6X-RT on v1.17 sends HEARTBEAT from compid 1 (PX4) and from compid 236
with autopilot=MAV_AUTOPILOT_INVALID. connect() accepted the first
heartbeat from anyone and left target_component at pymavlink's default of
0, so param and mission requests went out as broadcast, the other
component could answer, and param_stress mixed its PARAM_VALUE stream
(index, count, values) into the autopilot's download.
connect() now waits for a heartbeat whose autopilot is not
MAV_AUTOPILOT_INVALID and pins target_system/target_component to its
source. pymavlink only latches target_system once and never sets the
component, so the pin holds for the life of the connection. The
wait_heartbeat() helper shares that filter and, once pinned, only counts
the pinned component; wait_reconnect() goes through connect() and gets
the same behavior after a reboot.
PARAM_VALUE replies in px4bench.params, param_stress and link_forwarding
are filtered to the pinned component, and flight_mission uses the pinned
component for MAVFTP and the armed/disarmed heartbeat checks instead of a
hardcoded 1.
Diagnosed by @farhangnaderi in #27852.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
log_transfer crashed with a raw ImportError on setups whose pymavlink
predates the mavftp module (#27840). mavftp first shipped in pymavlink
2.4.42, so raise the floor in pyproject.toml, Tools/setup/requirements.txt
and the README, and guard the import in px4bench.ftp so an old install
reports a clear upgrade hint instead of a traceback. Both mavftp
consumers now import it through the guard.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
Running the suite from main against a release/1.17 FMUv6X-RT board
(#27840) failed boot_health and flight_mission at 'shell open' while
reboot_loop passed the same check 10/10 in the same run. The failing
opens were the sessions immediately following a closed one, so the
single wake write was being consumed before the firmware nsh pipes were
reading and no amount of waiting could recover it. Re-send the wake
write every second until the deadline, strip stale BENCHOPEN wake lines
from subsequent command output, and replace the 5s hardcoded at every
call site with one px4bench.SHELL_OPEN_TIMEOUT (10s) so the budget is
tunable in one place.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
The --viewer path (tee MAVLink frames to UDP and enable the
HIL_STATE_QUATERNION/HIL_ACTUATOR_CONTROLS streams so Hawkeye could
render an SIH flight live) was never validated end to end: Hawkeye
never actually rendered from it. Rather than ship an unproven feature,
drop it.
Removes px4bench.attach_viewer_tee (and its function-local socket use),
the flight_mission --viewer/--viewer-port args, the viewer tee call and
the stream-enable block, and the now-unused --board-dev arg that only
fed those stream commands. The SIH flight logic, firmware gate,
storage/serial tests, and the arming gate are unchanged. README and the
bench testing docs page drop the Hawkeye section and all --viewer
mentions.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
A hardware run exhausted the board's task/fd table after a few probe
cycles: shell open failed while heartbeat still worked, and continued
writes eventually stalled the link. Not a firmware hang, a resource leak
the tooling drove.
The firmware spawns a new nsh task plus two pipes (four fds) on the
first SERIAL_CONTROL write of a session (Mavlink::get_shell) and only
frees them when a SERIAL_CONTROL message arrives with the RESPOND flag
cleared (Mavlink::close_shell). Two tooling bugs combined to leak them:
- MavlinkShell.open() returned True on any buffered byte, including
leftover output from a prior session, so it reported a live shell
without one. It now confirms an actual 'nsh>' prompt or a completed
echo-sentinel round-trip before returning True, keeping the same
timeout-and-report-failure behavior.
- open/close cycles reopened before the single flags=0 teardown was
processed, spawning a second shell before the first was freed. close()
now sends the flags=0 teardown and then briefly drains incoming
SERIAL_CONTROL until quiet (bounded, never a hang) so the firmware
runs close_shell before a caller can reopen.
Lifecycle: every MavlinkShell user now opens one session, runs all its
work on it, and closes it in a finally so no error path leaks a shell.
The SIH flight and storage tests run all their probes on a single shell
(shell_command_exists already takes an open shell); flight_mission no
longer opens a throwaway probe shell separate from the flight session
beyond the unavoidable reboot boundary in enter_sih.
Folds in the sentinel fix (run() sends the echo sentinels on their own
input line, because nsh aborts a ';' chain when a command fails, which
previously lost the sentinel for a failing probe and read as a stall).
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
The simulated flight is the only test that exercises commander,
navigator, and land-detector flight logic on real NuttX scheduling;
leaving it out of the default suite meant the most valuable coverage
was routinely skipped. It now runs last in the sequence.
Arming safety gets two explicit paths: on a TTY the operator sees a
prominent warning that the flight controller WILL ARM (simulated
flight, pwm_out_sim replaces the real outputs, board must be bare) and
must type 'arm'; declining records a skip, not a failure. For
automation and manufacturing lines --allow-arming bypasses the prompt,
and a non-TTY run without it skips the flight with a warning naming
the flag.
Before touching any configuration the test probes the live firmware
for the SIH module ('simulator_sih status'; NuttX nsh replies
'nsh: <cmd>: command not found', apps/nshlib/nsh_parse.c) and skips
with a warning when the module is absent.
Shared plumbing in px4bench: EXIT_SKIP (75) recorded as SKIP by the
orchestrator, shell_command_exists() for probe-and-skip, and
arming_gate() used by both the orchestrator flow and the standalone
flight test. Config mutation by the flight test is expected; it
restores the original airframe and SYS_HITL afterwards.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
Today's runs executed against whatever firmware happened to be on the
board (an old April build at first) and nothing checked. For a
qualification tool that is unacceptable: before any test starts the suite
now establishes, and can control, exactly which build it is testing.
New px4bench/firmware.py: .px4 metadata parsing and validation (fields
verified against Tools/px_mkfw.py and px4_uploader.py), board identity
via ver all over the nsh shell, flashing through Tools/px4_uploader.py
with streamed output and a hard timeout, HW-arch to build-target
inference by enumerating boards/, an early wrong-board check against
firmware.prototype board_id, and the named firmware_identity check
(git-hash prefix match against the artifact).
run_bench_suite.py preflight supports four firmware sources, all
converging on the same flash + verify path: keep what is on the board
(--any-firmware), flash a local file (--firmware), build the inferred
target from this source tree (--build, --target, --build-timeout), or
download a GitHub release artifact (--release TAG|latest, via gh;
asset naming verified against the v1.17.0 release). --expect-hash
verifies without flashing. Flags are mutually exclusive; with none
given, a TTY gets an operator menu covering the same four sources, and
automation exits with an error before touching the board. The detected
identity is printed, written to firmware.json in the suite report dir,
and stamped into every test's report dir via PX4BENCH_FIRMWARE_INFO.
If the board's mavlink is wedged the uploader cannot soft-reboot it into
the bootloader (hit live today); after repeated reboot attempts the gate
prints an operator instruction to replug USB so the uploader catches the
bootloader at power-on.
sih/flight_mission.py gains a verify-only --expect-hash and stamps the
firmware identity into its report dir; it never flashes.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
Turn the flat script pile into a standalone project so contributors can
navigate and extend it:
- px4bench/ shared library package: core primitives in __init__ (Reporter,
connect, MavlinkShell, reboot/replug, viewer tee, mavlink status parsers,
pymavlink add_message workaround) plus protocol modules params.py,
missions.py, and ftp.py extracted from the tests. Zero helper duplication
remains across scripts.
- bench/ holds the real-firmware tests (boot_health, reboot_loop,
usb_replug, link_forwarding, param_stress, mission_stress, log_transfer);
sih/ holds the simulated flight (flight_mission), making the
simulation/no-simulation boundary explicit.
- pyproject.toml (px4bench 0.1.0, BSD-3-Clause, pymavlink/pyserial deps,
pyulog extra) so pip install -e Tools/bench_test works; every script
remains directly runnable without installation via a parent-dir path shim.
- README rewritten contributor-first: architecture, per-test justification
tied to the v1.18 risk areas, why pymavlink over MAVSDK, how to add a
test, baseline workflow.
- Consistent CLI surface (shared connection args; --report-dir replaces
log_transfer's --outdir); decorative section banners removed.
All hardware-learned behavior is preserved exactly: param echo drain and
match-by-value, shell sentinel strip-all, the add_message workaround,
explicit param save before reboot, mission clear before upload, RTL in
MAV_FRAME_MISSION, and the post-flight ULog download.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>