log_transfer crashed with a raw ImportError on setups whose pymavlink
predates the mavftp module (#27840). mavftp first shipped in pymavlink
2.4.42, so raise the floor in pyproject.toml, Tools/setup/requirements.txt
and the README, and guard the import in px4bench.ftp so an old install
reports a clear upgrade hint instead of a traceback. Both mavftp
consumers now import it through the guard.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
The --viewer path (tee MAVLink frames to UDP and enable the
HIL_STATE_QUATERNION/HIL_ACTUATOR_CONTROLS streams so Hawkeye could
render an SIH flight live) was never validated end to end: Hawkeye
never actually rendered from it. Rather than ship an unproven feature,
drop it.
Removes px4bench.attach_viewer_tee (and its function-local socket use),
the flight_mission --viewer/--viewer-port args, the viewer tee call and
the stream-enable block, and the now-unused --board-dev arg that only
fed those stream commands. The SIH flight logic, firmware gate,
storage/serial tests, and the arming gate are unchanged. README and the
bench testing docs page drop the Hawkeye section and all --viewer
mentions.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
The flight test is now part of the default suite; both the bench README
and the bench testing guide said it ran separately. Update the sequence
description, the risk map and suite tables (storage_stress and
flight_mission rows, serial_loopback for fixtures), the SIH sections
(probe-and-skip, the typed 'arm' confirmation, --allow-arming for
automation), the production section (--allow-arming on the line,
serial_loopback with a jumper on the fixture), and the firmware gate
--build description (bench capability report with variant and config
line recommendations). Storage threshold defaults are justified where
the flags are documented.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
Today's runs executed against whatever firmware happened to be on the
board (an old April build at first) and nothing checked. For a
qualification tool that is unacceptable: before any test starts the suite
now establishes, and can control, exactly which build it is testing.
New px4bench/firmware.py: .px4 metadata parsing and validation (fields
verified against Tools/px_mkfw.py and px4_uploader.py), board identity
via ver all over the nsh shell, flashing through Tools/px4_uploader.py
with streamed output and a hard timeout, HW-arch to build-target
inference by enumerating boards/, an early wrong-board check against
firmware.prototype board_id, and the named firmware_identity check
(git-hash prefix match against the artifact).
run_bench_suite.py preflight supports four firmware sources, all
converging on the same flash + verify path: keep what is on the board
(--any-firmware), flash a local file (--firmware), build the inferred
target from this source tree (--build, --target, --build-timeout), or
download a GitHub release artifact (--release TAG|latest, via gh;
asset naming verified against the v1.17.0 release). --expect-hash
verifies without flashing. Flags are mutually exclusive; with none
given, a TTY gets an operator menu covering the same four sources, and
automation exits with an error before touching the board. The detected
identity is printed, written to firmware.json in the suite report dir,
and stamped into every test's report dir via PX4BENCH_FIRMWARE_INFO.
If the board's mavlink is wedged the uploader cannot soft-reboot it into
the bootloader (hit live today); after repeated reboot attempts the gate
prints an operator instruction to replug USB so the uploader catches the
bootloader at power-on.
sih/flight_mission.py gains a verify-only --expect-hash and stamps the
firmware identity into its report dir; it never flashes.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
The README opened by framing the whole suite around one merged PR
series. The value is general: CI never boots NuttX and SITL runs on a
host OS, so defects in boot ordering, link lifecycle, storage, loop
rates, and RTOS-only concurrency are invisible until a board is on a
bench. Reword the introduction around that gap and the timeout-first
design rule instead of the originating incident.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
Turn the flat script pile into a standalone project so contributors can
navigate and extend it:
- px4bench/ shared library package: core primitives in __init__ (Reporter,
connect, MavlinkShell, reboot/replug, viewer tee, mavlink status parsers,
pymavlink add_message workaround) plus protocol modules params.py,
missions.py, and ftp.py extracted from the tests. Zero helper duplication
remains across scripts.
- bench/ holds the real-firmware tests (boot_health, reboot_loop,
usb_replug, link_forwarding, param_stress, mission_stress, log_transfer);
sih/ holds the simulated flight (flight_mission), making the
simulation/no-simulation boundary explicit.
- pyproject.toml (px4bench 0.1.0, BSD-3-Clause, pymavlink/pyserial deps,
pyulog extra) so pip install -e Tools/bench_test works; every script
remains directly runnable without installation via a parent-dir path shim.
- README rewritten contributor-first: architecture, per-test justification
tied to the v1.18 risk areas, why pymavlink over MAVSDK, how to add a
test, baseline workflow.
- Consistent CLI surface (shared connection args; --report-dir replaces
log_transfer's --outdir); decorative section banners removed.
All hardware-learned behavior is preserved exactly: param echo drain and
match-by-value, shell sentinel strip-all, the add_message workaround,
explicit param save before reboot, mission clear before upload, RTL in
MAV_FRAME_MISSION, and the post-flight ULog download.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
sih_flight.py switches the board to a SIH airframe (SYS_HITL=2, physics
on the FMU), flies takeoff, a 3-waypoint square, and RTL as an auto
mission, asserting arming, takeoff, waypoint progression, landing, and
auto-disarm with per-phase timeouts, then restores the original config.
Covers the flight-logic paths the bench tests cannot reach; pwm_out_sim
replaces real outputs so nothing is driven on the rails.
--viewer tees every received MAVLink frame to UDP (frame per datagram,
same framing as the SITL viewer channel) and enables the
HIL_STATE_QUATERNION/HIL_ACTUATOR_CONTROLS streams on the board, so
Hawkeye renders the flight live from a serial-connected board.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
Semi-automated bench tests for qualifying v1.18 on real Pixhawk-class
hardware after the TSAN/concurrency series (#27606, #27809, #27813).
CI builds NuttX but never boots it, and the failure mode of this class
of regression is a silent hang, not a crash: every operation here has a
hard timeout and a timeout is reported as FAIL naming what stalled.
Tests map to the reworked subsystems: boot_health (WorkQueue/uORB
snapshot with baseline diffing), dual_link_forwarding (nested-send lock
path under simultaneous two-link load), param_torture (set/readback and
reboot persistence on SDLOG_UTC_OFFSET), mission_torture (dataman and
the mission shared-state mutex, 220 items, link alternation),
mavftp_log (logger on/off plus MAVFTP download and ULog verification),
reboot_loop and the operator-assisted usb_replug (mavlink instance and
RAM lifecycle across re-enumeration). run_bench_suite.py orchestrates
the non-interactive tests with a per-test watchdog.
Requires pymavlink/pyserial, pyulog optional. Nothing arms the vehicle.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>