* feat(motor_failure_detector): add current-residual detection library
Per-motor model-based motor-failure detector for multirotors. Predicts the
expected ESC current from the commanded actuator signal
(I_exp = current_slope_a*u + current_idle_a) and trips on a low-pass-filtered
residual held past a trip band for a persistence time. The command (not measured
rpm) is the fault-independent reference: a dead motor that draws no current while
still commanded produces a large residual, whereas a current-vs-rpm model would
mask a clean stop.
Band and persistence are configurable per residual direction. Undercurrent is the
fault-consistent side -- an open circuit, a shed propeller or a desync makes a
motor draw less than commanded -- while overcurrent covers a winding short,
mechanical jam or demagnetization. They are separate because the benign residual
is far from symmetric: a fast throttle increase draws acceleration current the
static model cannot predict, so overcurrent excursions are large and unbounded,
whereas an undercurrent residual is floored at -I_exp. A single band sized to
reject the overcurrent transients can therefore end up wider than the largest
undercurrent fault a motor can physically produce, which makes a full current
collapse undetectable rather than merely slow. Set undercurrent_threshold_a <= 0
to keep one symmetric band on the residual magnitude.
Built on PX4 primitives rather than hand-rolled equivalents: update() takes a
timestamp (esc_status.timestamp in flight, the log timestamp in replay) and
derives dt internally; the residual filter is a mathlib AlphaFilter; the
persistence debounce is a systemlib::Hysteresis per direction (with a sticky latch
on top, and reset across data gaps so a dropout can't complete a pending trip).
The data-gap threshold (kMaxGap = 0.3 s) is sized above the ~10 Hz arming-check
feed interval and below the 400 ms ESC-offline timeout: a live feed is never
mistaken for a gap (a smaller value equal to the feed interval reset the filter
and debounce on the majority of jittered ticks, so a sustained fault never
survived the persistence window -- offline replay: 0/100 sampling phases latched
vs 100/100 gap-free), while a genuine outage still resets and is handled by the
separate ESC liveness guard.
Includes an 18-case gtest spanning the failure-mode spectrum and edge cases:
healthy-never-trips (incl. high throttle), open-circuit and overcurrent trips,
severe prop-loss undercurrent, the missed ~50% partial loss (documents the
detection floor), the per-direction bands (a tighter undercurrent band catching a
collapse one symmetric band misses, band independence, per-side persistence),
persistence (short spike crosses-threshold-but-recovers), gap reset, the 10 Hz
feed vs. real-gap boundary, dropout-holds, multi-motor, the exclusion gate,
monitor-only, and reset/latch.
* refactor(motor_failure_detector): resolve the undercurrent band at configure time
An unset undercurrent threshold now mirrors the overcurrent band in
configure(), so update() runs the same two bands in every configuration and
no longer carries a second threshold test that reads like a repeat of the
disable gate. A symmetric band is the two-band case with equal bands.
Also name esc_status CONNECTED_ESC_MAX at kMaxMotors so the constant is
findable, use the PX4 time literals for the persistence conversions, and
drop the excluded temporary.
* feat(commander): use current-residual detector for the ESC motor-failure check
Replace the flat, instantaneous current check in EscChecks::checkMotorStatus with
the MotorFailureDetector library (added in the parent PR). Per motor it predicts
the expected ESC current from the commanded thrust
(I_exp = MOTFAIL_C2T*u + MOTFAIL_IDLE) and trips on the low-pass-filtered
residual, with an independent band and hold time per direction: MOTFAIL_UNDER held
for MOTFAIL_UND_TIME below the expectation, MOTFAIL_OVER held for MOTFAIL_OVR_TIME
above it.
Because the two directions need separate bands, the single band and hold time are
renamed to name the direction they act on: MOTFAIL_OFF -> MOTFAIL_OVER and
MOTFAIL_TIME -> MOTFAIL_OVR_TIME, migrated in param_translation.cpp. The old band
applied to both directions, so it carries over to the overcurrent side, and
MOTFAIL_UNDER defaults to 0, which keeps a single symmetric band -- an existing
vehicle behaves as before. MOTFAIL_IDLE also defaults to 0, and the whole check
stays gated by FD_ACT_EN (off by default).
The detector is driven off esc_status.timestamp, so a telemetry stall freezes the
filter instead of integrating stale current; the separate 400 ms ESC telemetry
timeout stays in checkEscOnline(). It is reconfigured from parameters while
disarmed, so a parameter change never resets a latched failure mid-flight. The
motor_failure_mask still feeds failure_detector_status via
FailureDetector::publishStatus, leaving the ControlAllocator reaction path
untouched.
The mask reports the detector's sticky latch alone, not whether the motor is
currently evaluable. That preserves the invariant the previous check spelled out
-- "Only set, never clear mid-air: stopping the motor in response could make it
appear healthy again" -- because handling a failure makes exactly that motor
unevaluable: the control allocator drops it, which publishes a NAN command,
which excludes it from the detector. Re-deriving the mask from the exclusion
state clears it, the allocator restores all motors, and the next check re-trips.
Measured in SIH on a hexarotor with a missing propeller: 83 mask transitions in
nine seconds, with a reversed recovery motor swinging between idle and near-full
reverse at the 10 Hz check rate. An ESC going offline after a real failure
retracts the report the same way. The latch is only ever set on a tick where the
motor was evaluable, so reporting it alone cannot report a motor that was never
judged.
* feat(motor_failure_detector): add offline ulog replay tool
Add a host tool that replays a recorded .ulg through the real MotorFailureDetector
and prints the per-motor trip verdict, for validating and tuning the detector
offline (EKF2-replay style: the real library drives recorded data, not a
re-implementation).
Built on ulog_cpp (PX4's C++ ULog library, added as a git submodule and built for
the posix host target only), so it reads ESC current, motor command and arming
state by field name from the log's own format -- one binary replays logs of any
PX4 message version with no rebuild. The feed mirrors the runtime in escCheck:
evaluated only while armed, at the runtime's 10 Hz cadence, with the per-ESC
current-report startup latch and reset-on-disarm. Config comes from the log's
MOTFAIL_* parameters with a built-in calibration fallback, and --set
MOTFAIL_NAME=value overrides any knob for what-if retuning. Because the runtime's
exact 10 Hz sample phase cannot be reconstructed offline, a single pass over the log
drives several detector instances on staggered grid offsets and reports a per-motor
peak band and trip fraction rather than a single phase's verdict, flagging when the
result is phase-dependent.
batch_replay.sh runs the tool over a folder of logs and summarizes each verdict
(OK / FAIL / SKIP) from its exit code.
Off by default so it never enters the firmware build or CI: it is a posix-only,
EXCLUDE_FROM_ALL host executable gated behind the MFD_REPLAY_TOOL cache option.
Build it with:
cmake -DMFD_REPLAY_TOOL=ON build/px4_sitl_default
ninja -C build/px4_sitl_default mfd_replay
* docs(motor_failure_detector): document the current-residual motor-failure check
Add a configuration page for motor failure detection, next to the other
actuator topics, and a developer page for the offline replay tool:
- what the check compares and why the under/overcurrent bands are separate
- the new MOTFAIL_* parameters, including the rename migration
- how to fit the current model and pick the bands from the vehicle's own
logs with mfd_replay / batch_replay.sh
The "Motor Failure Trigger" section in the failsafe page described the
pre-rename parameters, the unfiltered symmetric comparison and claimed the
failure was non-latching, so it is reduced to a summary of the trigger and
its action, pointing at the new page for the thresholds.
* docs(docs): Subedit for simpler introductions and terminology
* docs(docs): Improve upgrading from v1.18 section
* feat(motor_failure_detector): add current-model fitting tool
mfd_fit derives MOTFAIL_C2T and MOTFAIL_IDLE from recorded logs, so the model
and the thresholds are taken from the same data.
Like mfd_replay it reads ESC current, motor command and arming state through
ulog_cpp by field name, so one binary handles any log version. Each log is a
single streaming pass that accumulates only the sums a least-squares line needs,
so memory does not grow with the log or with the number of logs, and a whole
corpus can be fitted in one invocation.
Alongside the parameters it reports the residual spread the trip bands have to
clear, and each motor's bias against the shared model.
* docs(docs): document motor failure detection tuning and the fitting tool
Follow-up to review of the detection and replay pages:
- fix the link left behind by the Failure Response -> Failure Action rename
- drop the Multicopter badge: the check has no vehicle-type gate, so a VTOL's
multirotor motors and its forward-flight motor are both checked
- describe the model-fitting step without the statistics jargon
- state that only healthy logs may be used, for the fit and the thresholds
- document mfd_fit, and reduce the release note to a pointer
* feat(motor_failure_detector): report residual extremes and add a band sweep
The replay tool printed the largest residual magnitude, which hid whether an
excursion was above or below the model. Since the two trip bands are chosen per
direction, that left the reader running one band at a time to find out. It now
prints the signed minimum and maximum instead, so each band's headroom is
readable straight from the table.
Add sweep_matrix.sh alongside batch_replay.sh: it sweeps one band against its
hold time over a set of logs and prints how many logs false-trip at each pair,
so the lowest usable band per hold time is the first zero in a row.
Both tools now number motors from 1, matching the "Motor N ..." the vehicle
reports, rather than from 0.
* docs(docs): rework the motor failure calibration walkthrough
Follow a real calibration through both pages and fix what it ran into:
- the replay page's worked example is one vehicle end to end, with every
number and pasted output taken from that corpus
- the fitting tool, the single-log run and the band sweeps are ordered the way
the work is actually done, with replaying a whole fleet last
- the sweeps are shown as a matrix of false trips per (hold time, band), and
the grids are derived from the residual extremes rather than guessed
- commands match the output they produce, and say what they need: mfd_replay
without --set uses a fallback calibration that belongs to another vehicle
- vehicle_status is optional, not required, for the replay
- drop the repetition: batch_replay.sh, the hover-current ceiling and the
phase-sensitivity note were each explained two or three times
* fix(motor_failure_detector): separate configure and reset with the appropriate triggers
* refactor(motor_failure_detector): clarify leading edge hysteresis as trigger time compared to "persistence"
* fix(escCheck): encapsulate _motor_failure_direction handling into MotorFailureDetector
* fix(astyle): exclude ulog_cpp submodule form check
* fix(motor_failure_detector_tools): fail log replay when no log is present
* refactor(motor_failure_detector): Improve documentation
* fix(escCheck): check load per ESC not per motor function
This was already done in my original implementation. It supports checking multiple ESCs mapped to the same motor function individually. It's also consistent with other ESC checks that run for every ESC that reports telemetry.
* fix(motor_failure_detector): run update() per motor and consider individual telemetry timestamps
* fix(motor_failure_detector): simplify output interface to getters of actually used results
* fix(motor_failure_detector): inline what doesn't need to be separate
* fix(motor_failure_detector): make the bands independent and drop special cases
- Each trip band is now independent: a threshold <= 0 disables only that band. The undercurrent band no longer copies the overcurrent one.
- Drop the MOTFAIL_OFF/MOTFAIL_TIME parameter migration. The new parameters start at their defaults.
- Treat a non-finite current like a non-finite command (not evaluated, filter and debounce reset). No driver reports NaN current, so the path only existed for its test.
- Compute the residual as max(current, 0) - I_expected. Same as before, but written such that it's obvious: regeneration counts as 0A.
- Drop unnecessary trigger time parameter clamp to >= 0 before converting to unsigned microseconds, metadate already decalres that as out of range.
- Count each ESC sample once by its own timestamp in mfd_fit, like the motor_failure_detector.
* fix(motor_failure_detector): check all still considered online ESC telemetry
* fix(motor_failure_detector): declare all link dependencies
* fix(motor_failure_detector_tools): reuse log reading logic to reduce duplication
* fix(motor_failure_detector): replace reversible flag getting passed into the detector for separate handling
Having a NAN command already handles that.
* test(escChecks): add unit tests for load based failure detection
---------
Co-authored-by: Hamish Willee <hamishwillee@gmail.com>
Co-authored-by: Matthias Grob <maetugr@gmail.com>
Co-authored-by: Andrea Bernasconi <andrea.bernasconi@auterion.com>
* build(mavsdk_tests): stop building the C++ integration tests
The integration tests are moving to the native MAVSDK Python package, so
the C++ test binary and its pinned libmavsdk-dev package are no longer
needed. The C++ sources stay in the tree as a reference until all tests
are ported.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>
* test(mavsdk_tests): run the integration tests with MAVSDK Python
The MAVSDK pip package is now a native binding of the C++ library, so
the tests can be written in Python without a C++ build or a matching
libmavsdk-dev package, and the version is pinned in requirements.txt.
The existing runner keeps starting PX4 for each case; only the test
executable changes to pytest. Cases are pytest node IDs and the Catch2
tags become pytest markers.
The tester is a port of the C++ AutopilotTester core. Checks done in
telemetry callbacks are collected and fail the test at teardown, since
MAVSDK Python swallows exceptions raised in callbacks.
The first ported test is takeoff and hold position.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>
* test(mavsdk_tests): port the multicopter tests to Python
Ports missions, offboard, manual flight, failsafes, failure injection,
RTL with rally points and geofence avoidance, home yaw and parameter FTP.
The tester gains the raw mission, manual control, failure, FTP and
MavlinkDirect helpers these need, plus an RTL subclass for rally points
and the geofence breach monitor.
The failsafe tests are parametrized over the height source instead of
being duplicated.
The parameter FTP download test now waits for the baro GNSS calibration
before comparing, since the offset it writes could otherwise change
between the PARAM_VALUE and the FTP download.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>
* test(mavsdk_tests): port the VTOL and control allocation tests to Python
Ports the VTOL missions, the VTOL return tests, figure eight, loiter
reposition, airspeed failure, the SIH XVert transition and the
gazebo-classic control allocation tests.
The loiter and figure-eight testers are merged into one loiter tester
that sends its commands as COMMAND_INT through MavlinkDirect. NaN fields
are sent as JSON null, since NaN is not valid JSON and MAVSDK reads null
as NaN.
Timeouts that were host time in C++ (approaches, figure eight) are now
sim time with values that fit the speed factors used in CI.
The gazebo runner can now take the world from the config, since pytest
node IDs no longer carry "(name.world)".
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>
* test(mavsdk_tests): settle the hover before injecting motor failures
The altitude and speed waits both pass while the vehicle is still
slowing down at the top of the climb, so the altitude reference for the
failure was taken mid-climb. At 30x speed the following overshoot was
just over the 1 m tolerance.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>
* test(mavsdk_tests): port the follow-me tests to Python
The C++ follow-me tests were never built: they were disabled pending a
MAVSDK change that has long since been merged, and the RC adjustment
test did not even compile. They now run with the other multicopter
tests.
Running them for the first time showed a few test bugs: target
locations were sent before follow-me was active, manual control was
dropped after 20 s, and the angle check ignored the rate limit from
FLW_TGT_MAX_VEL. The RC checks now measure the change from right before
each stick input.
The follow distance adjusts at about 40% of the nominal speed, since
PX4 only grows the setpoint while the drone is within 1 m of it. The
test accepts that for now.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>
* test(mavsdk_tests): remove the C++ integration tests
All tests now run with MAVSDK Python, so the C++ tests, the tester
classes and the bundled Catch2 are no longer needed.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>
* build(setup): install the integration test requirements
The MAVSDK and pytest pins move to Tools/setup/test-requirements.txt,
which the setup scripts and the dev container now install, so the
integration tests work after the normal environment setup. The mavsdk
pin allows 4.0.x patch releases only.
The file lives in Tools/setup because that is the dev container's build
context. Container builds that stage only requirements.txt skip it.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>
* docs(test_and_ci): describe the Python MAVSDK integration tests
Covers the setup, running test cases by pytest ID, running a test
against a PX4 instance started by hand, and writing tests with the
testers and fixtures. Drops the Catch2 notes and the outdated list of
test cases.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>
* refactor(packaging): port the SIH container mission test to MAVSDK v4
Since v4, the mavsdk package is the native binding with a different API,
so the script failed to import with the version the setup scripts now
install. It now uses the synchronous API.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>
* test(mavsdk_tests): work around MAVSDK-Python unsubscribe crash
Up to MAVSDK-Python 4.0.6, unsubscribe_*() releases the callback
trampoline before the C unsubscribe returns, so a callback that is
running or waiting for the GIL can jump into freed memory. This crashed
the hex test in CI. Keep the subscription entries until the tester is
closed, until the pin includes the fix.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>
* test(mavsdk_tests): retry the param FTP comparison until values settle
Waiting for CAL_BARO0_OFF to change does not work everywhere: at the
CI home location the baro already matches GNSS and the calibration
writes an offset of 0. Instead, compare the PARAM_VALUE and FTP
downloads and retry a few times while parameters still settle.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>
* test(mavsdk_tests): port the vision target tests to Python
Port the precision landing tests from #29265: the pad is simulated with
TARGET_RELATIVE and TARGET_ABSOLUTE sent through MavlinkDirect, and the
estimate is checked from LANDING_TARGET.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>
* test(mavsdk_tests): fail when a filter or case selects nothing
A config filter that matched no test, for example because of a typo,
ran zero cases and passed. Also, --case used fnmatch, which reads the
brackets of parametrized test IDs as a character class, so such a test
could not be selected by its ID. Treat brackets literally and fail when
nothing is selected.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>
* test(mavsdk_tests): check the COMMAND_ACK of sent commands
Commands sent through MavlinkDirect were sent once and the ACK was
ignored, so a dropped or denied command only showed up later as an
unrelated timeout. Retry until acked like MAVSDK does, and check the
result.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>
* test(mavsdk_tests): check the condition once more on poll timeout
Also make the comment of check_rtl_approaches say that it can't fail.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>
* test(mavsdk_tests): disable baro auto-calibration in the param FTP test
Port #29359 to the Python test: the baro GNSS calibration can save a
new CAL_BARO0_OFF between the PARAM_VALUE and FTP downloads. Disable it
instead of retrying the comparison.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>
---------
Signed-off-by: Julian Oes <julian@oes.ch>
The Micro-XRCE-DDS-Client-v3 submodule was added without a corresponding
prune entry, so 'make format'/'make check_format' would reformat its
vendored sources. Add it to the exclusion list alongside the existing
Micro-XRCE-DDS-Client entry.
* feat(astyle): add make format_changed for diff only style fixes
`make format` is pretty slow because it always considers the entire
source tree. For developing on top of a clean state, considering only
files that differ from HEAD should be sufficient.
* docs(astyle): document new format_changed target
Several helper scripts assumes bash is available at /bin/bash. That breaks on systems
such as NixOS, where bash is resolved from PATH instead of a fixed /bin location and
causes failures like `bad interpreter` during `make format`, e.g., on my host machine:
```sh
$ make format
/PX4-Autopilot/Tools/astyle/check_code_style.sh: /PX4-Autopilot/Tools/astyle/fix_code_style.sh: /bin/bash: bad interpreter: No such file or directory
```
This change switches these entrypoints to `#!/usr/bin/env bash` so they locate bash properly.
No functional changes intended.
Signed-off-by: Onur Özkan <work@onurozkan.dev>
* always reschedule from measurementReadyCallback, don't update range mode is processMeasurement fails, reschedule trigger faster if Argus_TriggerMeasurement fails, refactor mode switching logic, update perf counters
* make format
* make format exclude microstrain submodule
---------
Co-authored-by: Jacob Dahl <dahl.jakejacob@gmail.com>
From https://github.com/google/fuzztest.
This will now also add gtest (via cmake FetchContent)
And requires newer cmake (container updates):
CMake 3.19 or higher is required. You are running version 3.16.3
Note: Some files are autogenerated and I did not yet
go through each source of generation to fix it there.
Instead I adjusted the filter to only exclude those
such that we can fix things in steps.
- upstream libuavcan was broken and then marked deprecated, this fully absorbs the submodule (renamed libdronecan to deconflict) up to our last good working commit and all commit history is kept
- fixes https://github.com/PX4/PX4-Autopilot/issues/23727 (regression introduced in #23113)
- this puts us in a much better position to evolve the library as needed now that we have full control
- Added offset to Posix hrt time to account for synchronization with Qurt hrt time
- Added new Kconfig to configure synchronization of HRT timestamps on VOXL2
- Moved voxl2 libfc sensor library submodule from muorb module to boards directory
- Added check to make sure hrt_elapsed_time can never be negative
Previously uORB queue size was an awkward mix of runtime configurable (at advertise or IOCTL before allocate), but effectively static with all queue size settings (outside of test code) actually coming from the topic declaration (presently ORB_QUEUE_LENGTH in the .msg). This change finally resolves the inconsistency making the queue size fully static.
Additionally there were some corner cases that the muorb and orb communicator implementation were not correctly handling. This PR provides fixes for those issues. Also correctly sets remote queue lengths now based on the topic definitions.
* Made setting of uORB topic queue size in based on topic definition only
* Fixes to the ModalAI muorb implementation
* Removed libfc sensor from format checks
* msg/TransponderReport.msg ORB_QUEUE_LENGTH 8->16 (was set to higher in AdsbConflict.h
---------
Co-authored-by: Eric Katzfey <eric.katzfey@modalai.com>
Co-authored-by: Daniel Agar <daniel@agar.ca>
Reduces flash usage by ~16KB.
- compress formats at build-time into a single string with all formats
- then at runtime iteratively decompress using
https://github.com/atomicobject/heatshrink
- update all msgs to be directly compatible with ROS2
- microdds_client improvements
- timesync
- reduced code size
- add to most default builds if we can afford it
- lots of other little changes
- purge fastrtps (I tried to save this multiple times, but kept hitting roadblocks)
- mavlink/mavlink is now directly included as a submodule instead of the generated mavlink/c_library_v2
- this also switches to mavlink development.xml by default
I removed the filtering logic from the shell script in #18482 because
the new pre-commit hook already takes care of it.
The problem is if you don't update the .git/hooks/pre-commit file and
use the new shell script there's no filtering
of files done and it checks all files for every file.
This commit restores backwards compatibility because it does not hurt
until I have an automatic way to update the pre-commit hook file.
- sending protocol
- uorb event message & template methods for argument packing
- libevents submodule to send common events and handle json files
- cmake maintains a list of all (PX4) source files for the current build
(PX4 modules + libs), which is used to extract event metadata and
generate a json file
to suppress the warning message:
`xargs: warning: options --max-args and --replace/-I/-i are mutually exclusive, ignoring previous --max-args value`
when running make format.