The --viewer path (tee MAVLink frames to UDP and enable the
HIL_STATE_QUATERNION/HIL_ACTUATOR_CONTROLS streams so Hawkeye could
render an SIH flight live) was never validated end to end: Hawkeye
never actually rendered from it. Rather than ship an unproven feature,
drop it.
Removes px4bench.attach_viewer_tee (and its function-local socket use),
the flight_mission --viewer/--viewer-port args, the viewer tee call and
the stream-enable block, and the now-unused --board-dev arg that only
fed those stream commands. The SIH flight logic, firmware gate,
storage/serial tests, and the arming gate are unchanged. README and the
bench testing docs page drop the Hawkeye section and all --viewer
mentions.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
The full-suite persistence check failed on hardware with 'after reboot
SDLOG_UTC_OFFSET = 740 (expected 777)'. Firmware persistence is solid;
this was a test race. A PARAM_SET propagates through the param system
asynchronously, and phase_persistence ran 'param save' on the strength
of the PARAM_VALUE echo alone. When save raced ahead of the commit it
persisted the PRIOR value (740, the last value Phase 2 wrote). The echo
was not even proof of the new value: wait_param_echo returned
seen=[740, 777], a stale queued PARAM_VALUE from Phase 2 having slipped
past drain_param_values.
Make persistence deterministic with two gates instead of timing:
1. After setting the marker, read it back from the board until it reports
MARKER_VALUE (read_until) before saving. This confirms the value is
committed to RAM and trusts the board's read over any stale echo.
2. After 'param save', confirm the saved state with 'param show <name>'
and require the '+' (saved) flag AND the marker value before rebooting
(retry save once first). '*' means unsaved; the test never reboots on
an unsaved marker. Flag columns per src/systemcmds/param/param.cpp:822
(x used, + saved, * unsaved, l locked).
phase_set_readback now also trusts read_until over the echo, so the same
stale-queued-echo does not fail it either; its pass criteria stay the
readback match. Restores in phase_persistence and the final cleanup use
read_until too.
New shared helpers in px4bench.params: read_until(mav, name, expected,
timeout) -> (ok, last_seen), and param_is_saved(shell, name) plus its
parse_param_show() parser for the 'param show' saved flag and value.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
A hardware run exhausted the board's task/fd table after a few probe
cycles: shell open failed while heartbeat still worked, and continued
writes eventually stalled the link. Not a firmware hang, a resource leak
the tooling drove.
The firmware spawns a new nsh task plus two pipes (four fds) on the
first SERIAL_CONTROL write of a session (Mavlink::get_shell) and only
frees them when a SERIAL_CONTROL message arrives with the RESPOND flag
cleared (Mavlink::close_shell). Two tooling bugs combined to leak them:
- MavlinkShell.open() returned True on any buffered byte, including
leftover output from a prior session, so it reported a live shell
without one. It now confirms an actual 'nsh>' prompt or a completed
echo-sentinel round-trip before returning True, keeping the same
timeout-and-report-failure behavior.
- open/close cycles reopened before the single flags=0 teardown was
processed, spawning a second shell before the first was freed. close()
now sends the flags=0 teardown and then briefly drains incoming
SERIAL_CONTROL until quiet (bounded, never a hang) so the firmware
runs close_shell before a caller can reopen.
Lifecycle: every MavlinkShell user now opens one session, runs all its
work on it, and closes it in a finally so no error path leaks a shell.
The SIH flight and storage tests run all their probes on a single shell
(shell_command_exists already takes an open shell); flight_mission no
longer opens a throwaway probe shell separate from the flight session
beyond the unavoidable reboot boundary in enter_sih.
Folds in the sentinel fix (run() sends the echo sentinels on their own
input line, because nsh aborts a ';' chain when a command fails, which
previously lost the sentinel for a failing probe and read as a stall).
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
The flight test is now part of the default suite; both the bench README
and the bench testing guide said it ran separately. Update the sequence
description, the risk map and suite tables (storage_stress and
flight_mission rows, serial_loopback for fixtures), the SIH sections
(probe-and-skip, the typed 'arm' confirmation, --allow-arming for
automation), the production section (--allow-arming on the line,
serial_loopback with a jumper on the fixture), and the firmware gate
--build description (bench capability report with variant and config
line recommendations). Storage threshold defaults are justified where
the flags are documented.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
The firmware already ships sd_bench, sd_stress, and serial_test; the
suite was not exploiting them.
bench/storage_stress.py joins the default sequence before log_transfer
(storage health first, then the test that depends on storage). It runs
sd_bench with data verification and parses write/read throughput and
fsync latency into named checks with generous FAIL floors (defaults
justified in the README: 100 KB/s write, 200 KB/s read, 500 ms fsync;
a working-but-slow card prints a WARNING instead of flapping), then
sd_stress for file churn with a short default iteration count. Both
phases probe-and-skip independently: a firmware without a command or a
board without an SD card records SKIP with a warning, never FAIL.
bench/serial_loopback.py is a standalone operator/fixture test for
manufacturing: with a TX-RX loopback jumper installed on a chosen UART,
serial_test transmits a known pattern and the session summary
(rx/tx/rx err counts) plus pattern-error lines decide the verdict. Not
in the default sequence because it needs a physical fixture.
The firmware gate's --build path now prints a bench capability report
from one shared table: after reading default.px4board plus the label
overlay (labels merge both, cmake/kconfig.cmake:47-54) it lists whether
the image will contain simulator_sih, sd_bench, sd_stress, and
serial_test and which suite tests will run or skip. For anything
missing it names a sibling variant that has it (exact --target) and
the exact config line, and interactively offers to append the line to
the local board config, stating that the edit is local and uncommitted.
All four options are 'default n' in Kconfig, so the merged .px4board
lines decide presence.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
The simulated flight is the only test that exercises commander,
navigator, and land-detector flight logic on real NuttX scheduling;
leaving it out of the default suite meant the most valuable coverage
was routinely skipped. It now runs last in the sequence.
Arming safety gets two explicit paths: on a TTY the operator sees a
prominent warning that the flight controller WILL ARM (simulated
flight, pwm_out_sim replaces the real outputs, board must be bare) and
must type 'arm'; declining records a skip, not a failure. For
automation and manufacturing lines --allow-arming bypasses the prompt,
and a non-TTY run without it skips the flight with a warning naming
the flag.
Before touching any configuration the test probes the live firmware
for the SIH module ('simulator_sih status'; NuttX nsh replies
'nsh: <cmd>: command not found', apps/nshlib/nsh_parse.c) and skips
with a warning when the module is absent.
Shared plumbing in px4bench: EXIT_SKIP (75) recorded as SKIP by the
orchestrator, shell_command_exists() for probe-and-skip, and
arming_gate() used by both the orchestrator flow and the standalone
flight test. Config mutation by the flight test is expected; it
restores the original airframe and SYS_HITL afterwards.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
--target px4_fmu-v6xrt_bench previously skipped the early board_id
check because only bare vendor_model targets were known; a label
selects a .px4board config within the same board, so strip it and
resolve to the board dir. The uploader still enforces board_id against
the bootloader at flash time.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
New page in Platform Testing and CI covering px4bench: why on-hardware
verification exists alongside SITL and CI, the firmware traceability
gate, the bench test inventory, the SIH hardware-in-the-loop flight
test with Hawkeye, baseline comparison for upgrade regression and
golden-unit workflows, production end-of-line usage, and CI
integration. Linked from the section index and the sidebar.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
Today's runs executed against whatever firmware happened to be on the
board (an old April build at first) and nothing checked. For a
qualification tool that is unacceptable: before any test starts the suite
now establishes, and can control, exactly which build it is testing.
New px4bench/firmware.py: .px4 metadata parsing and validation (fields
verified against Tools/px_mkfw.py and px4_uploader.py), board identity
via ver all over the nsh shell, flashing through Tools/px4_uploader.py
with streamed output and a hard timeout, HW-arch to build-target
inference by enumerating boards/, an early wrong-board check against
firmware.prototype board_id, and the named firmware_identity check
(git-hash prefix match against the artifact).
run_bench_suite.py preflight supports four firmware sources, all
converging on the same flash + verify path: keep what is on the board
(--any-firmware), flash a local file (--firmware), build the inferred
target from this source tree (--build, --target, --build-timeout), or
download a GitHub release artifact (--release TAG|latest, via gh;
asset naming verified against the v1.17.0 release). --expect-hash
verifies without flashing. Flags are mutually exclusive; with none
given, a TTY gets an operator menu covering the same four sources, and
automation exits with an error before touching the board. The detected
identity is printed, written to firmware.json in the suite report dir,
and stamped into every test's report dir via PX4BENCH_FIRMWARE_INFO.
If the board's mavlink is wedged the uploader cannot soft-reboot it into
the bootloader (hit live today); after repeated reboot attempts the gate
prints an operator instruction to replug USB so the uploader catches the
bootloader at power-on.
sih/flight_mission.py gains a verify-only --expect-hash and stamps the
firmware identity into its report dir; it never flashes.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
The README opened by framing the whole suite around one merged PR
series. The value is general: CI never boots NuttX and SITL runs on a
host OS, so defects in boot ordering, link lifecycle, storage, loop
rates, and RTOS-only concurrency are invisible until a board is on a
bench. Reword the introduction around that gap and the timeout-first
design rule instead of the originating incident.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
Turn the flat script pile into a standalone project so contributors can
navigate and extend it:
- px4bench/ shared library package: core primitives in __init__ (Reporter,
connect, MavlinkShell, reboot/replug, viewer tee, mavlink status parsers,
pymavlink add_message workaround) plus protocol modules params.py,
missions.py, and ftp.py extracted from the tests. Zero helper duplication
remains across scripts.
- bench/ holds the real-firmware tests (boot_health, reboot_loop,
usb_replug, link_forwarding, param_stress, mission_stress, log_transfer);
sih/ holds the simulated flight (flight_mission), making the
simulation/no-simulation boundary explicit.
- pyproject.toml (px4bench 0.1.0, BSD-3-Clause, pymavlink/pyserial deps,
pyulog extra) so pip install -e Tools/bench_test works; every script
remains directly runnable without installation via a parent-dir path shim.
- README rewritten contributor-first: architecture, per-test justification
tied to the v1.18 risk areas, why pymavlink over MAVSDK, how to add a
test, baseline workflow.
- Consistent CLI surface (shared connection args; --report-dir replaces
log_transfer's --outdir); decorative section banners removed.
All hardware-learned behavior is preserved exactly: param echo drain and
match-by-value, shell sentinel strip-all, the add_message workaround,
explicit param save before reboot, mission clear before upload, RTL in
MAV_FRAME_MISSION, and the post-flight ULog download.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
sih_flight.py switches the board to a SIH airframe (SYS_HITL=2, physics
on the FMU), flies takeoff, a 3-waypoint square, and RTL as an auto
mission, asserting arming, takeoff, waypoint progression, landing, and
auto-disarm with per-phase timeouts, then restores the original config.
Covers the flight-logic paths the bench tests cannot reach; pwm_out_sim
replaces real outputs so nothing is driven on the rails.
--viewer tees every received MAVLink frame to UDP (frame per datagram,
same framing as the SITL viewer channel) and enables the
HIL_STATE_QUATERNION/HIL_ACTUATOR_CONTROLS streams on the board, so
Hawkeye renders the flight live from a serial-connected board.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
First hardware run (fmu-v6xrt over USB, macOS) surfaced three issues:
- param_torture consumed PARAM_VALUE echoes positionally; one duplicate
broadcast desynced every subsequent check. Drain stale messages before
each set and match the echo against the expected value, reporting all
observed values so a genuine wrong-echo bug remains visible.
- the shell sentinel's second safety echo leaked into the next command's
captured output; strip any sentinel line, not just the current one.
- pymavlink 2.4.49 add_message() intermittently crashes with TypeError
when the first message of an instanced type arrives with its instance
field unset; wrap it defensively until fixed upstream.
Full suite now passes 5/5 against fmu-v6xrt.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
Semi-automated bench tests for qualifying v1.18 on real Pixhawk-class
hardware after the TSAN/concurrency series (#27606, #27809, #27813).
CI builds NuttX but never boots it, and the failure mode of this class
of regression is a silent hang, not a crash: every operation here has a
hard timeout and a timeout is reported as FAIL naming what stalled.
Tests map to the reworked subsystems: boot_health (WorkQueue/uORB
snapshot with baseline diffing), dual_link_forwarding (nested-send lock
path under simultaneous two-link load), param_torture (set/readback and
reboot persistence on SDLOG_UTC_OFFSET), mission_torture (dataman and
the mission shared-state mutex, 220 items, link alternation),
mavftp_log (logger on/off plus MAVFTP download and ULog verification),
reboot_loop and the operator-assisted usb_replug (mavlink instance and
RAM lifecycle across re-enumeration). run_bench_suite.py orchestrates
the non-interactive tests with a per-test watchdog.
Requires pymavlink/pyserial, pyulog optional. Nothing arms the vehicle.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
Since it's 1:1 reflecting a MAVLink message we better also keep the order otherwise it's less easy to follow.
The "flags" title in the status enum is misleading. Those aren't flags, there can only be one status at a time a uint8 would not even have 12 bits to set.
The per-channel send mutex is genuinely recursive: callers hold lock_send()
while the MAVLink C library re-enters send_start()/send_finish() on the same
mutex via MAVLINK_START_UART_SEND. On NuttX a recursive mutex only works with
CONFIG_PTHREAD_MUTEX_TYPES; without it pthread_mutexattr_settype() is a no-op
and the nested lock deadlocks. Only a handful of boards happened to enable the
option (for Zenoh), so mavlink could deadlock at startup on most boards, e.g.
when bringing up a USB link.
Enable CONFIG_PTHREAD_MUTEX_TYPES on the NuttX boards that build mavlink, add
a compile-time guard so a board missing the option fails the build loudly
instead of deadlocking silently, and check the settype() return. Enabling the
option is free in flash and RAM on these boards since they already enable
CONFIG_PRIORITY_INHERITANCE / CONFIG_PTHREAD_MUTEX_ROBUST.
Signed-off-by: Julian Oes <julian@oes.ch>
The mission manager's _shared_state_mutex was recursive only because the
count updaters (update_geofence_count / update_safepoint_count) called the
public update_active_mission(), which re-took the same lock. That is the
"public method calls another public locking method" antipattern, not a
genuine need for recursion.
Split the state update into an update_active_mission_locked() helper that
assumes the lock is held and returns the mission_s to publish. The count
updaters call it while already holding the lock and publish after unlocking;
the public update_active_mission() locks around the helper. The mutex is now
a plain, statically-initialized (PTHREAD_MUTEX_INITIALIZER) mutex, dropping
the pthread_once runtime init and the non-portable recursive attribute.
This removes the dependency on CONFIG_PTHREAD_MUTEX_TYPES for this mutex,
which is not enabled on most NuttX boards.
Signed-off-by: Julian Oes <julian@oes.ch>
* feat(FlightTaskAuto): add RC yaw override during Auto modes
* feat(FlightTaskAuto): disengage RC yaw nudge on flight-mode switch
* docs: document MPC_AUTO_RC_YAW in MC Auto mode pages
Describe the RC yaw nudge (MPC_AUTO_RC_YAW) on the Mission page.
Add parameter rows to the Hold and Land pages where MPC_LAND_RC_HELP takes
precedence over it for yaw, and document MPC_LAND_RC_HELP there too.
* feat(FlightTaskAuto): updated param description to warn against COM_RC_OVERRIDE disabled
The long description recommends to have a switch on the RC to change to Position mode
* fix(FlightTaskAuto): unified yaw nudging with new MPC_AUTO_NUDGING parameter
* docs(FlightTaskAuto): MPC_AUTO_RC_YAW and MPC_LAND_RC_HELP parameters were replaced by the unified MPC_AUTO_NUDGING bitmask
* fix(parameters): migrate MPC_LAND_RC_HELP to MPC_AUTO_NUDGING bit 1
* docs(1.18): move stick override and add nudging changes
---------
Co-authored-by: Matthias Grob <maetugr@gmail.com>
The test provides a guard against the _q_mc_to_fw quaternion changing,
and sanity checks a _copy of_ the conversion implemented in Run().
It cannot guard against breaking the implementation in Run() without
changing the quaternion - that is the job of integration tests.
- In tailsitter case, construct current quaternion from _R, which is
aready rotated correctly a bit above
- Downside: bloaty solution of converting to rotation matrix, adjusting
that, and converting back to quaternion
The AirBrainH743 breaks out a single I2C bus (I2C1) that also carries the
onboard DPS310 baro and IIS2MDC mag. Because the bus was marked internal,
the external-compass autostart (which scans external buses with -X) never
probed it, so an external GPS/compass on that bus was never bound.
Mark I2C1 external so the external-mag autostart scans it, and add a
px4_i2c_device_external() override (guarded by BOARD_OVERRIDE_I2C_DEVICE_EXTERNAL)
that keeps the onboard baro and mag classified as internal. This mirrors the
approach used on fmu-v6c. The onboard baro/mag are now started explicitly by
bus and address (-X -b 1) since -I no longer matches the external bus.
Also enable the i2cdetect command, which is handy for probing the shared
external bus on this board.
Signed-off-by: Julian Oes <julian@oes.ch>
On a moving deck the lift-off XY lock alone still leaned the vehicle: the
position smoother carried the inherited deck velocity into the takeoff and
braked it with a backward tilt during the thrust-limited ramp, which sprang
back at FLIGHT.
Give altitude priority for the first metre after lift-off: hold the live
horizontal position through the ramp (zero tracking error, no lean or spring
at FLIGHT) and the live horizontal velocity through the ramp and the first
metre of climb. Self-nulling on a static takeoff (no inherited velocity), so
the path is identical for static and moving takeoffs - no new param, no branch.
Signed-off-by: gguidone <gennaroguido2002@gmail.com>
Track the live horizontal position during the takeoff ramp and freeze it at
TAKEOFF_STATE_FLIGHT, overriding the position target (not just the smoother) so the
climb stays vertical over the lift-off point instead of running to the commanded
lat/lon once tilt authority opens at FLIGHT. Reset the lock on (re)activation.
Pin the smoother's horizontal position to the live position during the takeoff ramp so a moving platform doesn't stall the climb; released at TAKEOFF_STATE_FLIGHT.
Signed-off-by: gguidone <gennaroguido2002@gmail.com>
test_multiple_semaphores_waiting() spawns a worker thread per TestCase, then
advances virtual time in a loop and completes each case in check(). Since
check() early-returns while !_thread_ready, a slow-to-start worker thread can
miss the whole loop: the loop advances past the case's timeout before the thread
registers, so the case is never completed. ~TestCase() then fails
EXPECT_TRUE(_is_done) and destroys a still-joinable std::thread, which calls
std::terminate() and aborts the test. This was flaky (~50% locally, worse under
CI load).
Wait for every worker thread to be ready before advancing virtual time. Also
re-comment the per-iteration std::cout that was left enabled.
Signed-off-by: Julian Oes <julian@oes.ch>
When a multicopter enters Hold (AUTO.LOITER) below NAV_MIN_LTR_ALT the
documented behavior is to first climb to that minimum altitude. The
clamp was added in 2022 to setLoiterItemFromCurrentPosition (the
fixed-wing branch) but never applied to the sibling
setLoiterItemFromCurrentPositionWithBraking, which is the rotary-wing
Hold path selected at loiter.cpp:131. As a result, multicopters
silently held at the entry altitude.
Apply the same min-altitude clamp in the braking variant so the
rotary-wing path matches the fixed-wing one.
Verified in SITL (jMAVSim x500): with NAV_MIN_LTR_ALT=20 and Hold
engaged from a 5m hover, the vehicle now climbs to 20m as documented.
Fixes#27578
Signed-off-by: saad1551 <saad.ashraf.9094@gmail.com>
Switches from using VehicleControlMode to a specific setpoint type message
with reply.
Reasons:
- in case a VehicleControlMode was dropped (e.g. when the mode switched
setpoint type), no confirmation was returned, and resulting in wrong
controller flags.
- cleaner interface separation: external modes do not need to configure
(or know) which controller to run for a certain setpoint
- allows for external modes to check for compatibility: e.g. a mode using
fixed-wing setpoint types can now be rejected on a multicopter.
- allows for further extensions, like a setpoint timeout
This makes it a bit more effort to add a new setpoint type. Specifically,
setpoint_types.cpp needs to be extended when adding a new setpoint message.
PX4 internal modes also make use of the setpoint types. The information
flow is:
nav_state -> setpoint type -> vehicle control mode flags
It also adds a timeout to the setpoint config, but is not implemented
yet.
This changes the interface for external modes and thus the compatibility
version is increased.
